Unit 42 at Palo Alto Networks found that sshd was hacked in 184 minutes in a honeypot experiment conducted by the lab.

If you’re thinking I just turned on this machine, nobody knows it’s here that is a wrong assumption. These were brand new honeypots and they were compromised on average in 3 hours. (Observing Attacks Against Hundreds of Exposed Services in Public Clouds)

But, I have great passwords or I only use ssh keys or certificates to secure my credentials.

What if a new vulnerability is discovered?

You need to patch, fast. A significant number of high-risk vulnerabilities are exploited after disclosure. Around 25% of high-risk CVEs are exploited within just 24 hours of going public. (SC Media, ThreatsHub)

I don’t want to be under that sort of pressure when it comes to accessing my remote shells.

Hackers can begin exploiting vulnerabilities even faster, with some vulnerabilities being attacked within just minutes after proof-of-concept code is released (Cybersecurity News).

How is this possible? Two underlying technologies are used by the good guys, and the bad guys.

ZMap and Masscan can scan the entire IPv4 address space in a matter of minutes. Once a new CVE is disclosed, especially for a high-risk vulnerability, attackers can use these tools to quickly locate systems that are exposed and vulnerable. (An Internet-Wide View of Internet-Wide Scanning, ZMap: Fast Internet-wide Scanning and its Security Applications).

For example, if a CVE affects a service running on a specific port, attackers can use Masscan to scan the internet for that open port, identifying potential targets almost immediately after the vulnerability is announced.

Get an 800 pound gorilla to guard your resources

Here’s how to access your SSH servers from the Internet without exposing them to scanners.

Keep your SSH servers on private subnets.
Don't open the ssh port to any network at all.
Let the ssm-agent usher access to remote shells.

How does this work?

Instead of allowing incoming connections, the ssm-agent running on your EC2 instance connects out to the AWS Session Manager (SSM) service. This requires that your EC2 instance be authorized to connect to SSM via an instance profile. SSM then provides sessions to users through its API as authorized by your IAM policies. Basically, it's a zero trust mechanism for your remote shells.

This glosses over a few details in getting this set up. But, the good news is that many AMIs come with ssm-agent. Check out the documentation for the AWS SSM for the full detail.

If you have an EC2 instance authorized to connect to the SSM service (there's a good chance that you do if you use Amazon or Ubuntu AMIs), you can probably connect this way already.

If you're using an Amazon or Ubuntu AMI and you can't connect this way, you may only need to add instance permissions for Session Manager.

Here is an example using the aws cli to open a remote shell:

% aws --region us-east-1 ssm start-session \
    --target i-instanceid
 
Starting session with SessionId: …
$ whoami # this is a shell on the ec2 instance
ssm-user

I use this in a slightly different way so that my ssh command works with other tools like VS Code.

In my ~/.ssh/config, I have this:

Host i-*
    IdentityFile YOURPRIVATEKEYPATH
    User MyUSER
    ProxyCommand sh -c "~/.ssh/aws-ssm-proxy-sso.sh %h %p REGION AWS_PROFILE"

# Please note!
# Substitute MyUSER, YOURPRIVATEKEYPATH, REGION, AWS_PROFILE 
# with values in accordance with your environment.

This configuration allows me to log in as an MyUSER user rather than the ssm-user. In this case, my User is specified and my private key is explicitly selected with IdentityFile.

The contents of the script referenced in the ProxyCommand is here:

% cat ~/.ssh/aws-ssm-proxy-sso.sh
#!/bin/bash
HOST=$1
PORT=$2
REGION=$3
export AWS_PROFILE=$4
/usr/local/bin/aws --region $REGION \
   ssm start-session --target $HOST \
   --document-name 'AWS-StartSSHSession' \
   --parameters portNumber=$PORT

Now when I use my ssh command, I get a shell on the ec2 instance as my user.

% ssh myhost
myname@ip-10-0-1-1:~$

Safe and sound. No scanners will be knocking at your ssh port after this! Stay safe out there.

Your SSH Server Could Be Hacked in 184 Minutes - Here's What to Do If You Use AWS

The all-in-one writing platform.

Write, publish everywhere, see what works, and become a better writer - all in one place.