How to organize terraform modules for multiple environments?

Question

Every Terraform guide on the web provides a partial solution that is almost always not the real picture.
I get that, not everyone has the same infrastructure needs, but what worries me that the common scenario with:

1. multiple environments (dev, stage)
2. remote backend (s3)
3. some basic resources (bucket or ec2 instance)

isn't presented anywhere on a real example project.
I'm looking for just that, and in the meantime, I have researched and concluded that apart from those needs I also want:

4. to utilize modules
5. to NOT use workspaces, but rather a distinct directory-per-environment approach
6. to NOT use terragrunt wrapper

---

My current structure, which does not utilize modules - only root module:

infra/ ------------------------------ 'terraform init', 'terraform apply' inside here*  
     main.tf ------------------------ Sets up aws provider, backend, backend bucket, dynamodb table   
     terraform.tfvars   
     variables.tf ----------------- Holds few variables such as aws_region, project_name...

My desired structure folder tree (for a simple dev & staging simulation of a single bucket resource) is I think something like this:

infra/  
     dev/  
        s3/  
            modules.tf ------ References s3 module from local/remote folder with dev inputs   
     stage/  
        s3/  
            modules.tf ------ References s3 module from local/remote folder with stage inputs   

But what about the files from my previous root module? I still want to have a remote backend in the same way as before, just now I want to have two state files (dev.tfstate and stage.tfstate) in the same backend bucket? How would the backend.tf files look like in each subdirectory and where would they be? In s3/ folder or dev/ folder?

It's kind of confusing since I'm transitioning from root module 'terraform init' approach, to specific subdirectory 'terraform init', and it's not clear to me whether I should still have a root-module or another folder for example called global/ which I should consider my prerequisite which I should init at the beginning of the project and is basically leave alone from that point on since it created the buckets which dev/ and staging/ can reference?

One more question is: what if I have s3/ ec2/ ecr/ subdirectories inside each environment, where do I execute 'terraform plan' command? Does it traverse all subdirectories?

When I have the answers and a clear picture of this above, it would be great to improve it by DRYing it up, but for now, I value a more practical solution through example rather than just a theoretic DRY explanation. Thanks!

Hoekstra

You do not want different modules.tf for different stages, that is exactly NOT how your infrastructure is supposed to look like. Because now your stage no longer is related to your dev and what you develop on dev will not be the same thing tested on stage will not be the same thing released on prod. You only have one directory where you put all the infrastrucutre, all the modules, all the logic. This is parametrized by some variables, e.g. the ec2 instance size or log retention time, etc. Those variables will have different values depending on the env, that is the only thing that changes. – Bartels

And then you WILL need a wrapper, either terragrunt or some custom basic shell script. Technically you are not required to have a wrapper but the alternative is having to remember a lot of parameters for the terraform commands. – Bartels

I think you got the wrong impression, I am not looking to DRY up this. I'm not sure if my post gave you the impression that my modules.tf are different, at the moment I think they should be the same (for the reasons you mentioned). I specifically added comment "with dev inputs" to indicate that I only change these variables between environments. But again, that was not my question at all, my question is everything around that actually. I appreciate the comment but you tackled unrelated thing which i mentioned in my last paragraph. – Hoekstra

Does https://mcmap.net/q/427475/-terraform-state-management-for-multi-tenancy or https://mcmap.net/q/427476/-ideal-terraform-workspace-project-structure answer your question? – Medamedal

Your findings seem to be the exact opposite of what I would recommend. Terraform workspaces exist to solve the exact problem you are trying to solve. Why do you think workspaces won't work for you? – Pedicle

@MarkB Well at this point I'm looking for anything that works. I mainly avoid workspaces due to them being "hidden", anti-documenting feature, and non-separation of prod and non-prod environments. If you are saying that what I want is not doable, then I am open to suggestions. Do workspaces work well with modules? Do you have an example of a project? – Hoekstra

@Medamedal Not really. From that article I get that I can use symlinks as a DRY improvement, but that example does not use modules, and most of my questions were: "How do I 'terraform plan' or 'terraform apply' for the whole dev environment, "How do I set up backend configuration if I have my environments separated by folders" etc. – Hoekstra

How are workspaces "anti-documenting"? You should really play around with these features some I think, to get an idea of how they work. Workspaces and modules are two totally different things, that work perfectly fine together. Modules should be used to encapsulate a set of resources, like a VPC module, an EC2 module, etc. I highly recommend browsing (or even using) some of the open source modules in the Terraform registry to see how they work registry.terraform.io/browse/modules – Pedicle

@MarkB I have no trouble understanding how a module works. Heck I even created one s3 module on separate repo and I am aware of how to call it. Most of my questions come from separating stuff into subdirectories and then not knowing how to manage state from that point on. Simply I have trouble understanding how I should configure and manage state in this subdirectories fashion. MANY examples on web talk about how you “shouldn’t put everything in one place” but FEW talk about how your terraform commands should adapt to acompany that. – Hoekstra

I am not totally against workspaces, i just saw few users point out flaws which seemed realistic so i went in the other way. – Hoekstra

Answer 1

I realized as @MarkB suggested, that terraform workspaces are actually a solution to multi-env projects.

So my project structure looks something like this:

infra/
  dev/
    dev.tfvars
  stage/
    stage.tfvars 
  provider.tf
  main.tf
  variables.tf

main.tf references modules, provider.tf set's up the provider, backend.tf would set up the remote backend (yet to add), etc.

The 'terraform plan' in this configuration becomes 'terraform plan -var-file dev/dev.tfvars' where I specify the file with a specific configuration for that environment.

Hoekstra

What happens with the tfstate files? – Verlie

@Verlie You can store them in a remote backend (s3) for example. I also realized throughout several projects that it's best to NOT mix IaC provisioning of resources which are used to store state of another IaC provisioning. In other words - to extract (or to do it manually) creation of remote s3 bucket from the infra that will be stored inside of it. – Hoekstra

Answer 2

I work with terraform 5 years. I did a lot of mistakes with in my career with modules and environments. Below text is just share of my knowledge and experience. They may be bad.

Real example project may is hard to find because terraform is not used to create opensource projects. It's often unsafe to share terraform files because you are showing all vulnerabilities from your intrastructure

Module purpose and size

You should create module that has single purpose, but your module should be generic.

Example module

You can create bastion host module, but better idea is to create a module for generic server. This module may have some logic dedicated to your business problem like, CW Log group, some generic security group rules, etc.

Application module

Sometimes it is worth to create more specific module.

Let's say you have application, that requires Lambda, ECS service, CloudWatch alarms, RDS, EBS etc. All of that elements are strongly connected.

You have 2 options:

• Create separated modules for each above items - But then your application uses 5 modules.
• Create one big module and then you can deploy your app with single module
• Mix above solutions - I prefer that

Everything depends on details and some circumstances.

But I will show you how I use terraform in my productions in different companies.

Separated definitions for separated resurces

This is project, where you have environment as directories. For each application, networking, data resoruces you have separated state. I keep mutable data in separated directory(like RDS, EBS, EFS, S3, etc) so all apps, networking, etc can be destroyed and recreated, because they are stateless. No one can destroy statefull items because data can be lost. This is what i was doing for last few years.

project/
├─ packer/
├─ ansible/
├─ terraform/
│  ├─ environments/
│  │  ├─ production/
│  │  │  ├─ apps/
│  │  │  │  ├─ blog/
│  │  │  │  ├─ ecommerce/
│  │  │  ├─ data/
│  │  │  │  ├─ efs-ecommerce/
│  │  │  │  ├─ rds-ecommerce/
│  │  │  │  ├─ s3-blog/
│  │  │  ├─ general/
│  │  │  │  ├─ main.tf
│  │  │  ├─ network/
│  │  │  │  ├─ main.tf
│  │  │  │  ├─ terraform.tfvars
│  │  │  │  ├─ variables.tf
│  │  ├─ staging/
│  │  │  ├─ apps/
│  │  │  │  ├─ ecommerce/
│  │  │  │  ├─ blog/
│  │  │  ├─ data/
│  │  │  │  ├─ efs-ecommerce/
│  │  │  │  ├─ rds-ecommerce/
│  │  │  │  ├─ s3-blog/
│  │  │  ├─ network/
│  │  ├─ test/
│  │  │  ├─ apps/
│  │  │  │  ├─ blog/
│  │  │  ├─ data/
│  │  │  │  ├─ s3-blog/
│  │  │  ├─ network/
│  ├─ modules/
│  │  ├─ apps/
│  │  │  ├─ blog/
│  │  │  ├─ ecommerce/
│  │  ├─ common/
│  │  │  ├─ acm/
│  │  │  ├─ user/
│  │  ├─ computing/
│  │  │  ├─ server/
│  │  ├─ data/
│  │  │  ├─ efs/
│  │  │  ├─ rds/
│  │  │  ├─ s3/
│  │  ├─ networking/
│  │  │  ├─ alb/
│  │  │  ├─ front-proxy/
│  │  │  ├─ vpc/
│  │  │  ├─ vpc-pairing/
├─ tools/

To apply single application, You need to do:

cd ./project/terraform/environments/<ENVIRONMENT>/apps/blog;

terraform apply;

You can see there is a lot of directories in all environments. As I can see there are pros and cons of that tools.

Cons:

• It is hard to check if all modules are in sync
• Complicated CI
• Complicated directory structure especially for new people in the team, but it is logic
• There may be a lot of dependencies, but this is not a problem when you think about it from the beginning.
• You need to take care, to keep exactly the same environments.
• There is a lot of initialization required and refactors are hard to do.

Pros:

• Quick apply after small changes
• Separated applications and resources. It is easy to modify small module or small deployment for it without knowledge about overall system
• It is easier to clean up when you remove something
• It's easy to tell what module need to be fixed. I use some tools I wrote to analyze status of particular parts of infrastructure and I can send email to particular developer, that his infrastructure needs resync for some reasons.
• You can have different environments easier than in the monolith. You can destroy single app if you do not need it in environemnt

Monolith infrastructure

Last time I started working with new company. They keep infrastructure definition in few huge repositories(or folders), and when you do terraform apply, you create all applications at the same time.

project/
├─ modules/
│  ├─ acm/
│  ├─ app-blog/
│  ├─ app-ecommerce/
│  ├─ server/
│  ├─ vpc/
├─ vars/
│  ├─ user/
│  ├─ prod.tfvars
│  ├─ staging.tfvars
│  ├─ test.tfvars
├─ applications.tf
├─ providers.tf
├─ proxy.tf
├─ s3.tf
├─ users.tf
├─ variables.tf
├─ vpc.tf

Here you prepare different input values for each environment.

So for example you want to apply changes to prod:

terraform apply -var-file=vars/prod.tfvars -lock-timeout=300s

Apply staging:

terraform apply -var-file=vars/staging.tfvars -lock-timeout=300s

Cons:

• You have no dependency, but sometimes you need to prepare some environment element like domains, elastic IP, etc manually, or you need to have them created before terraform plan/apply. Then you have problem
• Its hard to do cleanup as you have hundreds resources and modules at the same time
• Extremely long terraform execution. Here it takes around 45 minutes to plan/apply single environment
• It's hard to understand entire environment.
• Usually you need to have 2/3 repositories if you keep that structure to separate networking,apps,dns etc...
• You need to do much more work to deal with different environments. You need to use count etc...

Pros:

• It's easy to check if your infrastructure is up to date
• There is no complicated directory structure...
• All your environments are exactly the same.
• Refactoring may be easier, because you have all resources in very few places.
• Small number of initialization is required.

Summary

As you can see this is more architectural problem, the only way to learn it, is to get more experience or read some posts from another people...

I am still trying to figure out the most optimal way and I would probably experiment with first way.

Do not take my advantages as sure thing. This post is just my experience, maybe not the best.

References

I will post some references that helped me a lot:

• https://www.terraform-best-practices.com/
• https://github.com/antonbabenko/terraform-best-practices-workshop

Cryptogenic

You are comparing apples and oranges here. The only difference really is wether you have different something.tf files for the different stages or you have different variable files for the stages. The cons for the monolith are basically all incorrect, you simply should not have one deployment for your entire company, but nothing is stopping you from having one repository for one deployable artefact for one team. You should have the different deployables split like in your "micromodules" approach but the different modules should be the same for each environment. – Bartels

Yes I have the same modules for all environments. And I do not aggre with your thesis. For monolith this is how I work, and I know few more companies which organise code that way, which is bad for me. But it is hard to say: you simply should not have one deployment for your entire company,, because even if it has a lot of disadvantages it can be usefull in some situations. Hovever yess, you can put multiple deployments in one repository, but the idea was to put all group of resources in single deployment vs multiple small deployments. – Cryptogenic

But how to manage different state files for different environments? – Terraterrace

Yes, how do you mange state in monolith setup? If you execute dev it will try delete prod stuff if it was already run before. – Dreg

Answer 3

I can share what we ended up doing for our Indeni Cloudrail service. Hope it'll help.

We created a folder with all the modules. Then, there's a module called "all" which basically calls the other modules (s3, acm, etc.) with the right parameters. The "all" modules has variables.

Then, there are environments. Each of them calls the "all" module with specific values for these variables.

This is the output of a "find" command on the root of the Terraform code (sorry it isn't prettier). I removed many of the files as they weren't needed to get the point across:

./common.tfvars
./terragrunt.hcl
./environments
./environments/prod
./environments/prod/main.tf
./environments/prod/terragrunt.hcl
./environments/prod/lambda.layer.zip
./environments/prod/terraform.tfvars
./environments/prod/lambda.zip
./environments/prod/common.tf
./environments/dev-john
./environments/dev-john/main.tf
./environments/dev-john/terragrunt.hcl
./environments/dev-john/terraform.tfvars
./environments/dev-john/common.tf
./environments/mgmt-dr
./environments/mgmt-dr/data.tf
./environments/mgmt-dr/main.tf
./environments/mgmt-dr/terragrunt.hcl
./environments/mgmt-dr/network.tf
./environments/mgmt-dr/terraform.tfvars
./environments/mgmt-dr/jenkins.tf
./environments/mgmt-dr/keypair.tf
./environments/mgmt-dr/common.tf
./environments/mgmt-dr/openvpn-as.tf
./environments/mgmt-dr/tgw.tf
./environments/mgmt-dr/vars.tf
./environments/staging
./environments/staging/main.tf
./environments/staging/terragrunt.hcl
./environments/staging/terraform.tfvars
./environments/staging/common.tf
./environments/mgmt
./environments/mgmt/data.tf
./environments/mgmt/main.tf
./environments/mgmt/terragrunt.hcl
./environments/mgmt/network.tf
./environments/mgmt/terraform.tfvars
./environments/mgmt/route53.tf
./environments/mgmt/acm.tf
./environments/mgmt/jenkins.tf
./environments/mgmt/keypair.tf
./environments/mgmt/common.tf
./environments/mgmt/openvpn-as.tf
./environments/mgmt/tgw.tf
./environments/mgmt/alb.tf
./environments/mgmt/vars.tf
./environments/develop
./environments/develop/main.tf
./environments/develop/terragrunt.hcl
./environments/develop/terraform.tfvars
./environments/develop/common.tf
./environments/preproduction
./environments/preproduction/main.tf
./environments/preproduction/terragrunt.hcl
./environments/preproduction/terraform.tfvars
./environments/preproduction/common.tf
./environments/prod-dr
./environments/prod-dr/main.tf
./environments/prod-dr/terragrunt.hcl
./environments/prod-dr/terraform.tfvars
./environments/prod-dr/common.tf
./environments/preproduction-dr
./environments/preproduction-dr/main.tf
./environments/preproduction-dr/terragrunt.hcl
./environments/preproduction-dr/terraform.tfvars
./environments/preproduction-dr/common.tf
./README.rst
./modules
./modules/secrets-manager
./modules/secrets-manager/main.tf
./modules/s3
./modules/s3/main.tf
./modules/cognito
./modules/cognito/main.tf
./modules/cloudfront
./modules/cloudfront/main.tf
./modules/cloudfront/files
./modules/cloudfront/files/lambda.zip
./modules/cloudfront/main.py
./modules/all
./modules/all/ecs.tf
./modules/all/data.tf
./modules/all/db-migration.tf
./modules/all/s3.tf
./modules/all/kms.tf
./modules/all/rds-iam-auth.tf
./modules/all/network.tf
./modules/all/acm.tf
./modules/all/cloudfront.tf
./modules/all/templates
./modules/all/lambda.tf
./modules/all/tgw.tf
./modules/all/guardduty.tf
./modules/all/cognito.tf
./modules/all/step-functions.tf
./modules/all/secrets-manager.tf
./modules/all/api-gateway.tf
./modules/all/rds.tf
./modules/all/cloudtrail.tf
./modules/all/vars.tf
./modules/ecs
./modules/ecs/cluster
./modules/ecs/cluster/main.tf
./modules/ecs/task
./modules/ecs/task/main.tf
./modules/step-functions
./modules/step-functions/main.tf
./modules/api-gw
./modules/api-gw/resource
./modules/api-gw/resource/main.tf
./modules/api-gw/method
./modules/api-gw/method/main.tf
./modules/api-gw/rest-api
./modules/api-gw/rest-api/main.tf
./modules/cloudtrail
./modules/cloudtrail/main.tf
./modules/cloudtrail/README.rst
./modules/transit-gateway
./modules/transit-gateway/attachment
./modules/transit-gateway/attachment/main.tf
./modules/transit-gateway/README.rst
./modules/transit-gateway/gateway
./modules/transit-gateway/gateway/main.tf
./modules/openvpn-as
./modules/openvpn-as/main.tf
./modules/load-balancer
./modules/load-balancer/outputs.tf
./modules/load-balancer/main.tf
./modules/load-balancer/vars.tf
./modules/lambda
./modules/lambda/main.tf
./modules/vpc
./modules/vpc/3tier
./modules/vpc/3tier/main.tf
./modules/vpc/3tier/README.rst
./modules/vpc/peering
./modules/vpc/peering/main.tf
./modules/vpc/peering/README.rst
./modules/vpc/public
./modules/vpc/public/main.tf
./modules/vpc/public/README.rst
./modules/vpc/endpoint
./modules/vpc/endpoint/main.tf
./modules/vpc/README.rst
./modules/vpc/isolated
./modules/vpc/isolated/main.tf
./modules/vpc/isolated/README.rst
./modules/vpc/subnets
./modules/vpc/subnets/main.tf
./modules/vpc/subnets/README.rst
./modules/guardduty
./modules/guardduty/README.md
./modules/guardduty/region
./modules/guardduty/region/main.tf
./modules/guardduty/region/guardduty.tf
./modules/guardduty/region/sns-topic.tf
./modules/guardduty/region/vars.tf
./modules/guardduty/.gitignore
./modules/guardduty/base
./modules/guardduty/base/data.tf
./modules/guardduty/base/guardduty-sqs.tf
./modules/guardduty/base/guardduty-lambda.tf
./modules/guardduty/base/variables.tf
./modules/guardduty/base/guardduty-kms.tf
./modules/guardduty/base/bucket.tf
./modules/guardduty/base/guardduty-sns.tf
./modules/guardduty/base/src
./modules/guardduty/base/src/guardduty_findings_relay.py
./modules/guardduty/base/src/guardduty_findings_relay.zip
./modules/jenkins
./modules/jenkins/main.tf
./modules/rds
./modules/rds/main.tf
./modules/acm
./modules/acm/main.tf

Vestige

Answer 4

Old article but thought I'd add my view as it's such a common question and there is no right or wrong approach (except to say that one massive deployment for ALL resources, that takes 20 minutes to figure out a Plan is asking for trouble as the blast radius would be huge). There's no hard rule for size of deployment, but I try to go with a rule of thumb of around 20-30 resources (max) and of course common sense. If it takes 10 minutes for TF to figure out the plan for adding a tag, then your deployment is probably too big.

After using Terraform for 4 or 5 years, I've tried all sorts, PowerShell wrappers, workspaces, terragrunt, pipelines & Terraform cloud. When using Open Source, I tend to go with an approach similar to @deltakroniker, using a different backend.tf file per environment as well as .tfvars. Run this from a pipeline to add approval gates etc and it works reasonably well, not perfect, but then what approach is?

It's similar to a workspace approach, except it allows you to specify different storage accounts for each env (when using Azure blob backend).

environments/
  dev/
    backend.tf
    environment.tfvars
  stage/
    backend.tf
    environment.tfvars 
tf-deploy/
  provider.tf
  main.tf
  variables.tf

plan or apply to an environment would be through command terraform plan --var-file=../environments/dev/environment.tfvars --backend-config=../environments/dev/backend.tf

Authentication to the backend is via environment variables (not in the backend.tf file). If done via a Pipeline then all sensitive vars can be gathered from a vault of some kind as part of the pipeline initialisation.

It's not perfect, you still have a question about how you try new module or provider versions, but don't want to promote to higher environments (with this approach, what you get in Dev, you ultimately get in Prod). In this case, approval gates and management of these type of changes becomes key. Alternatively, incorporating some kind of branched deployment for these type of changes could be an option.

Sitwell

Answer 5

There is official guide for module structure based on official style guide by terraform

https://developer.hashicorp.com/terraform/language/style#module-structure

Abel