on
Bulk renaming terraform resources
Background
Back in December last year I joined a project where there was no consistency in the local name given to terraform resources and datasources, some things were kebab case such as;
resource "azurerm_key_vault" "team-a-secrets" {
...
}
other resources were in snake case, for example;
resource "azurerm_storage_account" "team_a_files" {
...
}
I even found one instance where both - and _ were used in the same name.
While terraform doesn’t care whether you used kebab case or snake case in your names, I do like local consistency within the same the same codebase. Considering that the resource names, datasource names and argument names defined in providers are always snake case this is generally my preference also.
Last week we had a couple of “Yak days” at work where we are allowed to focus on the smaller maintenance tasks that rarely get prioritised against other work and I chose to shave this particular Yak.
The work to be done
Before Terraform 1.1 refactoring the local name of resources like this would have either triggered a recreate of the resource, not ideal for infrastructure containing state, or required manually removing and importing resources from the state which is risky if you make any mistakes.
Since Terraform 1.1 we can now use moved blocks to do this refactoring in a
safe manner. Correcting the kebab case example above would require the following
code changes.
moved {
from = azurerm_key_vault.team-a-secrets
to = azurerm_key_vault.team_a_secrets
}
resource "azurerm_key_vault" "team_a_secrets" {
...
}
When running terraform plan with this you get the following output in the terraform plan;
# azurerm_key_vault.team-a-secrets has moved to azurerm_key_vault.team_a_secrets
Once applied you can then remove the moved block.
Scripting creation of the moved blocks
After a couple of hours of manually refactoring a couple of modules I realised this would take me a week of work if I wasn’t able to automate the job in some way. That’s when I set to work writing a little script to make my job easier. Luckily the HCL language is easily parsable with standard CLI tooling.
We always start our scripts with a shebang so we know what interpreter it will run with and tell it to exit immediately on errors. Then I defined a variable to hold the current root module folder to work in as we have a monorepo and I didn’t want to break everything at once.
#!/usr/bin/env bash
set -e
SEARCH_FOLDER=./prod/base-infra/
Then we need to find all of the terraform files in the module root (using nulls as separators for safe piping to the next stage)
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0
Then for each file we found, extract all the lines that start with the string resource ". This is the resurce definitions.
xargs \
--null \
grep '^resource "'
Next we parse those lines with awk, specifying " as the delimiter. If the
fourth token in the line contains a - then we output a moved block for the
resource.
awk \
--field-separator=\" \
'$4 ~ "-" { printf "moved {\n from = %s.%s\n to = %s.%s\n}\n\n", $2, $4, $2, gensub("-", "_", "g", $4) }'
This output can then be redirected to a file for later use. I used a temp filename for now as we don’t want this file to be picked up by other commands later in the script.
The same strategy can be used for creating moved blocks for data sources and module references.
Changing the resource names
In order to change the resource names themselves in the terraform files I used
a similar strategy to create a sed script that I could run towards the end of
the script. This looks the same as the code above but the awk command changed
to output valid lines of a sed script.
awk \
--field-separator=\" \
'$4 ~ "-" { printf "s/resource \"%s\" \"%s\"/resource \"%s\" \"%s\"/g\n", $2, $4, $2, gensub("-", "_", "g", $4) }'
This produces a sed script that looks like the following
s/resource "azurerm_key_vault" "team-a-secrets"/resource "azurerm_key_vault" "team_a_secrets"/g
And references to the resources
Just changing the resource names isn’t much use if there are references to those resources in other areas of the terraform code for example
resource "azurerm_key_vault_secret" "db_password" {
key_vault_id = azurerm_key_vault.team-a-secrets.id
...
}
These can also be changed by extending the sed script
awk \
--field-separator=\" \
'$4 ~ "-" { printf "s/%s\\.%s/%s.%s/g\n", $2, $4, $2, gensub("-", "_", "g", $4) }'
Which generates the following line
s/azurerm_key_vault\.team-a-secrets/azurerm_key_vault.team_a_secrets/g
Putting it all together
These last two commands need to be repeated for data sources and modules too.
Then we can run the sed script performing an inline replacement for all
terraform files. Finally we can move the refactoring.tf file to the correct
location.
All of this together looks like this;
#!/usr/bin/env bash
set -e
SEARCH_FOLDER=./prod/base-infra/
# Generate moved blocks for resources
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^resource "' \
| \
awk \
--field-separator=\" \
'$4 ~ "-" { printf "moved {\n from = %s.%s\n to = %s.%s\n}\n\n", $2, $4, $2, gensub("-", "_", "g", $4) }' \
> \
$SEARCH_FOLDER/refactoring.tf-soon
# Generate moved blocks for data sources
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^data "' \
| \
awk \
--field-separator=\" \
'$4 ~ "-" { printf "moved {\n from = data.%s.%s\n to = data.%s.%s\n}\n\n", $2, $4, $2, gensub("-", "_", "g", $4) }' \
>> \
$SEARCH_FOLDER/refactoring.tf-soon
# Generate moved blocks for modules
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
|
xargs \
--null \
grep '^module "' \
| \
awk \
--field-separator=\" \
'$2 ~ "-" { printf "moved {\n from = module.%s\n to = module.%s\n}\n\n", $2, gensub("-", "_", "g", $2) }' \
>> \
$SEARCH_FOLDER/refactoring.tf-soon
# Generate sed for resource usage
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^resource "' \
| \
awk \
--field-separator=\" \
'$4 ~ "-" { printf "s/%s\\.%s/%s.%s/g\n", $2, $4, $2, gensub("-", "_", "g", $4) }' \
> \
$SEARCH_FOLDER/refactoring.sed
# Generate sed for resource definition
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^resource "' \
| \
awk \
--field-separator=\" \
'$4 ~ "-" { printf "s/resource \"%s\" \"%s\"/resource \"%s\" \"%s\"/g\n", $2, $4, $2, gensub("-", "_", "g", $4) }' \
>> \
$SEARCH_FOLDER/refactoring.sed
# Generate sed for data source usage
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^data "' \
| \
awk \
--field-separator=\" \
'$4 ~ "-" { printf "s/data\\.%s\\.%s/data.%s.%s/g\n", $2, $4, $2, gensub("-", "_", "g", $4) }' \
>> \
$SEARCH_FOLDER/refactoring.sed
# Generate sed for data source definition
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^data "' \
| \
awk \
--field-separator=\" \
'$4 ~ "-" { printf "s/data \"%s\" \"%s\"/data \"%s\" \"%s\"/g\n", $2, $4, $2, gensub("-", "_", "g", $4) }' \
>> \
$SEARCH_FOLDER/refactoring.sed
# Generate sed for module usage
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^module "' \
| \
awk \
--field-separator=\" \
'$2 ~ "-" { printf "s/module\\.%s/module.%s/g\n", $2, $4, $2, gensub("-", "_", "g", $4) }' \
>> \
$SEARCH_FOLDER/refactoring.sed
# Generate sed for module definition
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null \
grep '^module "' \
| \
awk \
--field-separator=\" \
'$2 ~ "-" { printf "s/module \"%s\"/module \"%s\"/g\n", $2, gensub("-", "_", "g", $2) }' \
>> \
$SEARCH_FOLDER/refactoring.sed
# Apply sed script
find \
$SEARCH_FOLDER \
-name "*.tf" \
-print0 \
| \
xargs \
--null
sed -i -f $SEARCH_FOLDER/refactoring.sed
# Rename refactoring.tf to correct name
mv \
$SEARCH_FOLDER/refactoring.tf-soon \
$SEARCH_FOLDER/refactoring.tf
I’m sure this script could be optimised or written prettier but it was a one-time job and worked well enough for what I needed.