Make use of Extended IT to bolster your IT Support Offering
Extend your IT

Access denied to AWS EC2 Ubuntu instance after snapshot restore [SSH Password Authentication]

I’ve been pulling my hair out for about a week on this one so now that I’ve sorted it I feel the need to write this up in case anyone has the same issue in the future. I know it’s not that common to logon to Ubuntu AWS Instances by using password authentication instead of key authentication, but that’s just how this server is setup and it’s out of my control.

So the scenario is, I’ve been testing the AWS EC2 snapshot backups for a company who’s entire infrastructure is in AWS, we’ve never ran test restores before so this was our first chance to document the whole process. They have a mix of AWS Linux, Ubuntu & Windows Machines and I have to say for all the other servers the Snapshots work incredibly well and we have had no issues, until we got to the FTP Server.

The FTP server has a slightly different setup to all the other instances, it’s the only non-windows machine that doesn’t use key authentication for SSH login (which is the default with all the AWS Unix / Linux instances) but uses password authentication instead, so when you connect you have to enter a password.

So I restore the FTP Server snapshot to a new AMI and then launch an instance from that, all the settings are the same, the only thing that has changed is the IP address. So I launch putty, edit the IP address of my FTP Profile, connect to the server, enter the password but then I get ‘access denied’.

I KNOW the password is correct, it works for the live server, so there is no reason it shouldn’t work for this backup restore. There’s very little I can do through the AWS interface to fix this, so over the course of 4 separate days I attach the restored disk to another Ubuntu disk, make a various changes in the sshd_config, I attempt to create a new snapshot, a new AMI and a new instance, I try creating a new Instance without a key-pair, none of these options work for me.

I really need to get this sorted so I connect the disk back up to another instance and at this point I’m just checking everything to do with SSH, authentication & user accounts, eventually I check the etc/shadow file, I compare the password on the new server and the old server only to find out the PASSWORD HAS CHANGED. Yep, in the process of restoring the snapshot an additional exclamation mark has been added to the front of the encrypted hash of the password.

Live Server (Correct Password):

Restored Server (Incorrect Password):

I can’t quite understand how this has happened, but the solution was to just remove the ! from the restored server’s password.

In Short: If you fail to login using a password to an AWS Ubuntu server following a snapshot restore, check the etc/shadow file for any differences in the password.

If you have any questions or need a consultant to take a look at your AWS setup contact me – [email protected]