What Terraform state is and why it is critical on AWS
Terraform state is the file (or set of files) that records the exact snapshot of your AWS infrastructure as Terraform deployed it. Every resource created, every configuration applied, every attribute assigned is stored there. Without this state, Terraform would not know which resources you have already created, how to modify them, or which ones to delete when your code changes.
This state is so critical on AWS that losing or corrupting it means Terraform no longer recognizes the resources it created itself. You end up with orphaned EC2 instances, RDS databases, and S3 buckets in your AWS account: billed but invisible to Terraform. Conversely, if you delete the state file by mistake, Terraform may decide to recreate the same resources because it thinks they do not exist, doubling both your infrastructure and your costs.
In teams that move past the artisanal stage on AWS (scale-ups and mid-sized companies spending more than a few thousand euros per month), state quickly stops being a local problem stored on a laptop. As soon as several engineers work on the same infrastructure, state becomes a shared artifact that must be locked (to prevent two simultaneous deployments from corrupting each other), backed up (to survive the loss of a machine), and versioned (to travel back in time when something goes wrong). This is why serious AWS teams store Terraform state on S3 with a remote backend rather than locally.
Configuring the S3 remote backend to secure state
By default, Terraform stores state in a local file named terraform.tfstate. This file contains your entire infrastructure in plain text, including passwords, API keys, and database details. Keeping it local works for a personal project or a prototype, but as soon as you work as a team on production infrastructure, it becomes a major risk.
The standard solution on AWS is to store state on a remote backend, typically an S3 bucket paired with a locking mechanism through DynamoDB. Here is how to set this up. First, create an S3 bucket dedicated to Terraform state, for example terraform-state-mycompany-prod. Enable versioning on this bucket so you can roll back to an earlier version of the state in case of corruption or error. Next, enable default encryption (Server-Side Encryption with an AWS-managed KMS key or a custom key) so nobody can read the raw contents of the state file stored on S3.
Also create a DynamoDB table named terraform-locks with a primary key called LockID (string type). This table lets Terraform lock the state while a deployment is in progress, preventing a second engineer from modifying the same state at the same time and corrupting it.
In your Terraform code, declare the backend as follows. Create a backend.tf file containing a terraform block with an s3 backend block. Specify bucket (the exact name of the S3 bucket), key (for example prod/terraform.tfstate to track the environment), region (the AWS region where you created the bucket), and dynamodb_table (the exact name of the DynamoDB table used for locking). Once this file is added and committed to your git repository, initialize Terraform with terraform init. Terraform detects the new backend configuration and automatically migrates your local state to S3.
For full control, restrict access to the S3 bucket and the DynamoDB table through IAM policies: only the AWS roles used by your CI/CD pipelines and your authorized developers should be able to read, write, and lock the state. Also block all public access to the S3 bucket using a public access block policy and private ACLs.
Detecting and understanding infrastructure drift
Despite all your efforts to manage infrastructure through Terraform, there is a permanent risk on AWS: someone modifying a resource directly through the AWS console, the AWS API, another script, or even an automated security patching script. An EC2 instance may have its security groups changed manually, an RDS may grow from 100 GB to 200 GB, an IAM policy may be modified to work around a restriction. These changes are not reflected in your Terraform code, creating a divergence between the Terraform state (which still believes the instance has its old configuration) and the actual AWS reality.
Terraform calls this drift. It is dangerous because it creates an illusion of control: your Terraform code looks intact and up to date, but the real infrastructure has drifted. If you apply a later change through Terraform, you risk accidentally overwriting the manual change, or getting unpredictable behavior.
To detect drift, use the terraform plan -refresh-only command. This command queries each AWS resource to see its actual current state, then compares it against what Terraform believes to be true (the state stored on S3). If a resource has changed outside of Terraform, terraform plan shows you the difference. For example, if an EC2 instance had three security groups in the state but currently has five in AWS (added by hand), you will see a line such as security_groups[3] = sgr-new-id and security_groups[4] = sgr-another-id in the plan.
Once drift is detected, you have several options. The simplest is to update your Terraform code to reflect reality: modify the resource block in your code to include the additional security groups, then run terraform plan again to confirm the plan is now empty. Alternatively, if the manual change was a mistake, you can use terraform apply to force the real infrastructure back to the state Terraform knows. Be careful: this operation can be destructive (for example, removing the manually added groups), so use it with caution and always after reviewing the plan.
In teams that genuinely automate their deployments, drift detection is often built into the CI/CD pipeline: a job scheduled every morning runs terraform plan -refresh-only and, if it detects drift, sends an alert to Slack or automatically creates an issue in the tracker. This ensures no manual change escapes the team's attention.
Managing multi-environment state without collisions
When you deploy infrastructure across several environments (development, staging, production), you have several Terraform state files to manage, one per environment. The natural temptation is to create several folders, terraform-dev, terraform-staging, terraform-prod, each with its own local terraform.tfstate file. This works while you are alone, but it quickly falls apart in a team.
The correct approach is to store a single Terraform codebase with variables that differ per environment, and to manage separate backends for each environment through a different S3 key. Take the backend.tf file again: instead of hardcoding the S3 bucket and key, use a separate backend configuration file per environment. Create backend-prod.tfvars containing bucket = terraform-state-mycompany and key = prod/terraform.tfstate. Create backend-dev.tfvars with key = dev/terraform.tfstate. When you initialize Terraform for production, run terraform init -backend-config=backend-prod.tfvars. For development, use terraform init -backend-config=backend-dev.tfvars.
This ensures that two engineers working simultaneously on dev and prod use completely different state files, stored in different locations in S3 and locked by different DynamoDB entries. No collision is possible. Also add Terraform variables for the resources that differ between environments: instance size, replica count, and so on. A terraform.tfvars-dev file contains environment = dev, instance_type = t3.small, replica_count = 1. A terraform.tfvars-prod contains environment = prod, instance_type = m5.large, replica_count = 3.
To avoid manual mistakes (for example, initializing with the wrong backend or forgetting to load the right variables file), automate everything in your CI/CD pipeline. Your Terraform job for prod should start with terraform init -backend-config=backend-prod.tfvars && terraform plan -var-file=terraform.tfvars-prod. The pipeline guarantees that you can never accidentally create a production resource in the development state.
Backing up and restoring state when something goes wrong
Even with the best practices in place, mistakes happen. An engineer may introduce a typo in a Terraform variable that accidentally deletes a production database, notices it too late, and has to restore the infrastructure to its state from an hour earlier. Or a state corruption leaves the file as invalid JSON, and terraform plan refuses to run. Without a backup strategy, this means panic and extended downtime.
Thanks to the S3 versioning you enabled on your state bucket, AWS automatically keeps a complete history of every version of the state file. Each time Terraform writes a new state after an apply, S3 records the previous version. If you catch an error a few minutes later, you can restore it easily.
To restore an earlier version of the state through the AWS console, go to the S3 bucket, click on terraform.tfstate, and look at the Versions tab. You will see a timestamped list of all versions of the file. Click the version ID that corresponds to the moment before the error, download that file, check its JSON contents to confirm it looks healthy, then re-upload it as the latest version of the terraform.tfstate file. Terraform will automatically pick up this restored version on the next terraform plan.
For an even safer and more automated approach, write a Python or Bash script that, before each production terraform apply, takes an explicit backup of the current state by copying it to an archive folder, for example s3://terraform-state-mycompany/backups/terraform.tfstate.2024-01-15T14h30m. One important point: never delete the old versions from your S3 bucket (avoid a lifecycle rule that would expire versions after N days). S3 storage for a state file, even with a year of history, costs a few euros per year. This backup cost is negligible compared to the risk of not being able to return to a healthy state.
In a team context, also document your state restoration procedure and test it once a year during a disaster recovery exercise. Too many teams discover their restoration procedure at the critical moment, when it fails. Testing it beforehand removes that surprise.
Automating drift detection in the CI/CD pipeline
Detecting drift manually every week is fragile: it depends on human vigilance and on remembering to run the terraform plan -refresh-only command. In teams that genuinely operate serious AWS infrastructure, drift detection is automated in the CI/CD pipeline.
The recommended practice is to schedule a daily CI/CD job (or every 6 hours for critical environments) that runs terraform plan -refresh-only for each environment. If the plan detects drift (that is, if there is a difference between the Terraform state and the AWS reality), the job sends a notification. This can be a Slack message with the details of the changes, an email to the SRE/DevOps team, or an issue automatically created in your tracker (Jira, GitHub Issues, and so on).
Here is an example of a GitLab CI or GitHub Actions job. In GitLab CI, add a job named detect-drift that runs every morning through a schedule: schedule cron: '0 8 *'. The job runs terraform init -backend-config=backend-prod.tfvars && terraform plan -refresh-only -no-color -out=tfplan. If terraform plan detects a difference, it returns a non-zero exit code. You can catch this exit code and send the plan details to Slack through a curl call to a Slack webhook, or archive the tfplan file as a job artifact for manual inspection.
In GitHub Actions, create a scheduled YAML workflow that runs a detect-drift job every 6 hours. After running terraform plan -refresh-only, use the actions/github-script action to automatically create an issue if changes are detected, or post a comment on a fixed issue or discussion.
The key benefit is visibility: you detect drift as soon as it appears, not after it has broken something or created a hidden inconsistency. The team receives an alert, discusses the cause (an accidental manual change, a security patch applied by a third party, or a deliberate change that was never committed), and decides: update the Terraform code to reflect reality, or roll back the manual change. This continuous feedback loop ensures that the Terraform code remains a faithful reflection of your AWS infrastructure.