Why secrets management must be strictly separated from your code and configuration
Even on small teams, one temptation surfaces again and again: leaving credentials in plaintext in the environment variables baked into the code or in a committed .env file. This practice creates a risk of massive exposure in the event of a repository leak, a screenshot, or historical access to version control. It also makes auditing impossible: who had access to which key, and when? Companies that have gone down this road end up becoming easy targets for attackers, who only have to scrape public or private repositories to find AWS tokens, Stripe keys, or certificates.
As soon as your deployment pipelines interconnect (pulling from GitHub, pushing to ECR, calling a database, sending a Slack notification), each step needs a distinct identity and a precise justification. If you grant your CI/CD pipeline a secret that gives access to ALL the secrets in the AWS account, a flaw in your CI/CD tool becomes a flaw in your entire system. The best practice is therefore to centralize secrets in a dedicated service (AWS Secrets Manager or Systems Manager Parameter Store), to restrict access through IAM with the principle of least privilege (each service receives only what it needs), and to automate injection at deployment time without ever letting the secret pass through logs or intermediate artifacts in plaintext.
The concrete benefit: in the event of a compromise, you can revoke a secret in seconds without redeploying the code, and you have full traceability of access. For scale-ups that are growing fast, this is the difference between a mature security culture and an infrastructure that accumulates security debt you will pay for dearly later.
AWS Secrets Manager, Systems Manager Parameter Store, and IAM: how the three fit together
AWS offers several building blocks for managing secrets and identities. AWS Secrets Manager is optimized for secrets that change regularly (database credentials, external API keys, certificates): it stores the secret, offers versioning, a rotation history, and webhooks to trigger actions (for example, updating the database) when a secret has rotated. Systems Manager Parameter Store (also called SSM) is more lightweight, without native rotation, but sufficient for static credentials or small secrets, with granular access through IAM and smooth integration into CloudFormation and Terraform.
The choice between Secrets Manager and Parameter Store depends on your automation load. If you work with RDS instances running on an auto-rotating credential, or if you manage Let's Encrypt certificates that expire regularly, Secrets Manager justifies its cost (USD 0.40 per secret per month) through its ability to orchestrate changes and notify the affected systems. For internal tokens or simple API keys, Parameter Store is enough and costs far less.
The key to how these three elements (Secrets Manager, Parameter Store, IAM) fit together lies in the IAM policies. You never give an IAM role the right to read ALL secrets: instead, you create a policy that authorizes your deployment pipeline (or your Kubernetes pod on EKS) to read ONLY the secret it needs, identified by a tag, a path, or a specific ARN name. For example, an ECS task that needs to connect to a PostgreSQL database receives an IAM role that authorizes only the secretsmanager:GetSecretValue action for the exact ARN of the secret containing the PostgreSQL password. If an attacker manages to take control of that task, they will not be able to access other secrets in the account.
This three-layer architecture (centralized storage, granular access control, automatic injection) is what separates robust secrets management from a hacked-together solution that will be bypassed as soon as delivery pressure rises.
Secure secret injection in your CI/CD deployment pipelines
Once your secrets are centralized in Secrets Manager or Parameter Store, the next challenge is to inject them at deployment time without exposing them in the logs or artifacts. The three most common patterns are injection as environment variables, injection into configuration files, and direct injection as a Docker or Kubernetes secret.
For a GitHub Actions or GitLab CI pipeline, the standard pattern is to retrieve the secret from Secrets Manager at the start of the deployment job, pass it as an encrypted environment variable to a deployment script (Terraform, CloudFormation, or shell script), and have it injected ONLY at the precise step where it is used (for example, the database connection) without ever displaying or storing it. GitHub Actions offers an automatic masking layer (Secrets Manager masks the values in the logs), but that is not enough: you must also lock down the repository so that only authorized pipelines and secure environments can access these variables.
In Kubernetes (EKS), the approach is to create Kubernetes Secrets that pull their content from AWS Secrets Manager through tools such as the ExternalSecrets Operator or the AWS Secrets and Configuration Provider (ASCP). Concretely, you define a Kubernetes Secret that says "go fetch the AWS secret named 'prod-db-password' and put its value here." When a pod starts, it has access to the secret without ever having to request it manually. IAM IRSA (IAM Roles for Service Accounts) guarantees that only authorized pods can access that secret.
For ECS (Elastic Container Service), you pass the secret through the secrets parameter of the task definition, which points to Secrets Manager or Parameter Store and injects the value directly as an environment variable inside the container, without it passing through the deployment logs or artifacts. The task itself needs an ECS task role with the appropriate IAM permissions to read the secret.
The critical point across all these patterns is the following: never write the secret into an intermediate artifact (a .env file, an unencrypted API response, a log file) that a party other than your target infrastructure could get hold of. Every step of your pipeline must treat the secret as sensitive data (encrypted in transit, never logged, never saved). The best teams add an audit trail: CloudTrail logs that record EVERY read of a secret, alerts if the secret is accessed from an unusual region or outside the planned deployment hours.
Identity management for cross-stage services: IAM, OIDC, and external OpenID Connect
Beyond static secrets (passwords, keys), managing dynamic IDENTITIES is just as critical. When your GitHub Actions pipeline needs to push an image to ECR, it cannot use a static AWS access key, nor share an AWS root secret across all your repositories. The modern solution is OpenID Connect (OIDC): GitHub Actions, GitLab CI, or any OIDC identity provider establishes a trust relationship with AWS through a federated IAM role.
Here is how it works in practice. You create a specific IAM role (for example, GitHubActionsDeployRole) whose assume-role policy accepts the OIDC tokens issued by GitHub for your organization and your repository. When your GitHub Actions pipeline runs, it requests an OIDC token from GitHub, then passes it to AWS while requesting to assume the role. AWS verifies the token (which only GitHub can issue), verifies that the repository and branch match what is authorized, and grants the pipeline a set of temporary, limited permissions. No access key is stored on GitHub; the identity is temporary and revoked at the end of the job.
This pattern eliminates two major sources of leakage: keys forgotten in the GitHub logs and static AWS keys saved outside of encryption. For teams using several CI/CD providers (GitHub Actions for code, ArgoCD for deployment, on-premise Jenkins for specialized builds), each can have its own federated IAM role with distinct permissions. A role for GitHub Actions can only deploy to staging environments, a role for ArgoCD in production has additional permissions only if deployed from a secure Kubernetes cluster, and so on.
To go further, some teams set up an internal temporary-credential issuing service (for example, HashiCorp Vault), which centralizes even more: a single source of truth for all secrets and identities, coordinated rotations, and revocation in one place. But for the majority of scale-ups, OIDC plus a federated IAM role with AWS plus restriction by branch and environment offers a very good security-to-complexity ratio.
Audit, rotation, and revocation: how you know your secrets are truly secure
Centralized secrets management is only secure if you monitor its usage. AWS CloudTrail records every call to Secrets Manager or Parameter Store: who read which secret, when, from which IP, through which IAM role. You can configure CloudWatch alerts to flag abnormal access (a read from an unusual region, a read of a secret not normally used, access outside deployment hours). For teams sensitive to compliance (GDPR, SOC 2, ISO 27001), these logs are mandatory: they prove that you have effective access control.
Secret rotation is the second pillar of monitoring. Secrets Manager can drive automatic rotation: you configure a Lambda that changes the password in the database and updates the secret in Secrets Manager, in parallel with itself. For simpler secrets that never change (external API keys you cannot rotate), you must at least document their known expiration date and set reminders at least three months before redeploying with a new key.
Revoking a compromised secret must be possible in under a minute. If an AWS key leaks, you must be able to disable it immediately, redeploy the services with a new key from Secrets Manager, and retroactively audit who used it and when. Modern teams even automate this step: a CloudTrail alert detects an abnormal secret, triggers a Lambda that invalidates the key, and triggers a redeployment of the EKS pods with a new key. All of this in a few seconds, with no manual intervention.
Finally, document your secret-access principles in a company policy or an ADR (Architecture Decision Record): who can read which secret, in which environments, through which service, via which request process? Teams without this policy end up with a layer cake of secrets that are never deleted ("just in case") and whose purpose no one remembers. Regular cleanup (every quarter) of unused secrets, confirmed by the CloudTrail logs, prevents this drift.
Integration with Terraform and Infrastructure-as-Code: declaring your secrets without exposing them
When you manage your infrastructure through Terraform or CloudFormation, you must declare that your application needs a secret without writing the secret itself in plaintext in the code. The standard Terraform pattern is to separate the declaration of the secret (creating an empty Secrets Manager resource) from its CONTENT (managed either manually, or through a rotation Lambda, or loaded from a secure source). Your Terraform code says: "create a secret named 'prod-db-password' with a dedicated KMS encryption key," but it never says "the value is ABC123."
A second, more common approach is to use the Terraform data source aws_secretsmanager_secret_version to READ an existing secret and inject it into a resource (for example, passing the password as an environment variable to an ECS task). This assumes the secret was created and populated by a process outside Terraform (a bootstrap Lambda, a SQL script, or a human intervention). Your Terraform then says: "read the secret named X from Secrets Manager and pass its value to the deployment." Terraform itself never sees the value in plaintext; only the Terraform state, unencrypted by default, contains references to the secret ARNs.
The major risk here is that the Terraform state gets pushed in plaintext to a repository or stored on an unencrypted backend. To avoid this: encrypt your Terraform state (S3 with server-side encryption plus DynamoDB for the lock), restrict read access to the AWS backend (IAM), and use Terraform remote state so that no one downloads the state locally. If you use Terraform Cloud, AWS State Store, or any other managed backend, verify that encryption and access are enabled by default.
A common anti-pattern: writing the secret into a .tfvars file, committing that file into the repository (even a private one), or passing it on the command line where it appears in the shell history. Instead, provide the secrets through TF_VAR_* environment variables, Terraform Cloud/Enterprise with UI-only sensitive variables, or an env management tool such as direnv or sops (Secrets OPerationS) that encrypts secret files with KMS or GPG.
When you work as a team, this means that one person (for example, a senior DevOps engineer) creates the secrets manually or seeds them through a Lambda when the AWS account is set up, and the rest of the team uses Terraform with a data source to READ and INJECT them, without ever seeing the value. A new team member can clone the repository, run terraform plan, and deploy without accessing the real secrets: their IDE only shows references, not passwords. This separation makes the code publicly shareable (as an open-source example) without risking a leak.
Auditing your current practices and concrete steps to migrate to a secure model
If you are reading this article, it is probably because you sense security debt in your secrets management. Here is how to diagnose your current situation and plan the migration without blocking your deployments.
Start with an audit of plaintext secrets. Grep through your code for hardcoded secrets, committed .env files, hardcoded variables in deployment scripts, and AWS keys stored on EC2 instances without an IAM role. Use tools such as TruffleHog or git-secrets to scrape your Git history and detect the patterns of AWS keys, SSH keys, and JWT tokens. Even if you remove a key from the latest commit, it remains in the Git history until a full branch rewrite. Document what you find: urgency (production? development?), current usage (who depends on this key?), and known exposure risk?
Second, audit your existing pipelines. How are secrets currently injected? Plaintext environment variables? Files passed via SCP? Credentials stored in Jenkins? Secrets in the logs? Trace the full flow: where the key starts, where it passes through, who has access at each step, and where it ends up. Note the bottlenecks (a secret shared across ten services, a single AWS key for all deployments).
Third, classify by impact and security. Identify the 5 to 10 most critical secrets (production database access, AWS keys for a critical S3 bucket, certificates) and start with THOSE. Migrate the production secrets that impact all deployments first, then the staging secrets, then the development secrets. For each secret, create a Secrets Manager resource, assign a dedicated KMS key, configure a restrictive IAM policy, and test the injection in a non-critical deployment before moving to production.
Fourth, update your pipeline. If you use GitHub Actions, replace the secrets stored in GitHub Settings with an OIDC connection to AWS, then retrieve the real secrets from Secrets Manager. If you use GitLab CI, the same approach applies: OIDC or ID tokens to AWS. For on-premise Jenkins or Terraform Cloud, use a credential-issuing service (AWS STS with OIDC, or an internal Vault).
Finally, establish continuous governance. Enable CloudTrail for all calls to Secrets Manager, set up a CloudWatch alert for every abnormal read, schedule a quarterly review of unused secrets, and document who can request access to which secret. The best teams automate the approval (a form that sends a Slack ticket, a manager approves it, a Lambda creates the secret and the IAM policy in 30 seconds).
The typical timeline for a scale-up: a full audit and priorities in 1 to 2 weeks, migration of the 5 to 10 critical secrets in 2 to 4 weeks (in batches of 1 to 2 secrets per sprint so as not to block deployments), and setting up governance and audit tooling in 2 additional weeks. Total: 1 to 2 months to move from a "plaintext secrets" culture to "centralized and audited management." It is a short, worthwhile investment that massively reduces the risk in the event of a breach and prepares you for future compliance audits.