How I Locked Myself Out of AWS (and What IAM Taught Me)
A botched inline policy, an expired access key, and an empty permission set — the IAM story every cloud engineer seems to live through once.
It was a Tuesday. I was tightening an IAM policy for our CI user — "least privilege" — and I attached a new inline policy before removing the old one. The new policy had a typo in the Action field: ec2:DesribInstances. AWS accepted the policy without complaint. The old admin policy I removed thirty seconds later did not come back without a fight.
Within a minute, our deployment user could do nothing. My own keys, which I'd been reusing "temporarily" for months, belonged to the same group. I was locked out of everything except the billing console.
I'm telling you this because IAM is the one AWS service where the failure mode is total, and everyone seems to learn it the same expensive way. Let's skip the expensive way.
The model in one paragraph
IAM has three moving parts: principals (users, roles, groups), policies (JSON documents that say allow or deny), and evaluation (explicit deny wins, then allow, then implicit deny — deny by default). Everything else is detail around those three ideas.
Roles matter more than users. Anything that runs — EC2 instances, Lambda functions, ECS tasks — should assume a role, not carry long-lived access keys. I now treat static access keys like production database passwords: rare, short-lived, rotated.
Permissions I actually give people
Instead of hand-writing policies for every human, I start from AWS managed job-function policies and restrict them:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:ap-south-1:123456789012:log-group:/app/*"
}
]
}
Scoped to one region, one log-group prefix, three actions. Boring policies are good policies.
Habits that would have saved me that Tuesday
MFA on everything, especially the root account. The root user bypasses all of IAM. I now set a calendar reminder every quarter to confirm root MFA and that no access keys exist for it.
Test policies before attaching. The IAM policy simulator is free and takes ninety seconds. If I'd typed DesribInstances into the simulator instead of production, I'd have caught it.
Keep one break-glass path. Our root user now has MFA, no keys, and the password is in a sealed envelope in a password manager — used exactly once since I wrote this rule, for an account-level lockout that IAM couldn't fix.
Give humans roles, assume them from the CLI. With AWS IAM Identity Center (or even a simple SSO setup), daily work happens through short-lived sessions:
aws sso login --profile admin
aws sts get-caller-identity
No long-lived keys sitting in ~/.aws/credentials waiting to be committed somewhere.
Use Access Analyzer. It flags policies granting access outside your account and resources that are effectively public. It found a bucket policy I'd copied from a tutorial in 2021 that allowed Principal: "*".
When you're already locked out
- Check if you have any other principal with permissions — sometimes a second admin user exists from initial setup.
- IAM Identity Center sessions sometimes survive an IAM user lockout.
- Root user recovery (email + password + MFA) is the official last resort; it can reset everything, but if you deleted the root access key, AWS support can't restore it either. Gone is gone.
- In my case, an org-management role from our SSO setup got me back in after twenty very quiet minutes.
The takeaway
Least privilege is not a policy you write once — it's a habit of shrinking permissions over time. Start with a permission set that works, then remove things every time nobody complains for a month. And never attach a policy you haven't simulated. That typo cost me a lunch break I'll never get back.