We’ve been connecting to servers with SSH for 30 years now. There’s no doubt that SSH is a solid protocol, safe to sit on a public IP and protect the server behind it by only allowing authenticated users. In a world full of services that seemingly go out of their way to leave backdoors, SSH has been solid. But, it’s also terrible.

Putting servers on the Internet for unfettered remote access makes them the target of continuous attacks. That SSH key for the former sysadmin that left 4 years ago still tucked away in an authorized_keys file that no one thought to remove? It still works and is probably still sitting on some personal laptop she owned back when she still worked for you. Worst of all, none of it is easily audited. Even with central syslogging, it’s difficult to know who’s been in your network, where they came from, and whether or not they should’ve been there.

We mitigated this problem for decades with the jump box (or bastion host, depending on age). But, it’s still just SSH. It’s still difficult to know if you made a configuration or access mistake. And most bastions are poorly set up. It’s rare to see a locked down, well run bastion.

VPNs are the most common way get remote admin access, but they have their own problems. And even once connected to the VPN, managing users and access centrally with SSH is suboptimal at best.

Fortunately, AWS Systems Manager offers a better way. Secure Session Manager (SSM) tunnels your SSH sessions through the AWS API, removing the need for direct connectivity to your VPC. Access is controlled by IAM, so IAM becomes your source of truth for access to both AWS and servers. If you’re using AWS SSO or an IAM SAML provider, your central directory is your source of truth for your entire operation.

SSM simplifies the management and connectivity to servers as well as internal endpoints like databases and admin interfaces. This comprehensive guide will walk you through the basics of AWS SSM, its operation, and how to use port forwarding to access private VPC resources.

Understanding AWS SSM

SSM operates through the AWS API and the SSM Agent, installed on your instances, allowing you to communicate via API to the service running on the instance. Once connected, SSM acts similar to SSH.

Because the connection is bridged through the AWS API, both client and server need access to it. Specifically, the client needs to be able to access the SSM API calls (which any PowerUser or AdministratorAccess user can – see below for more granular permissions if needed) and the server needs to be able to access a corresponding set of SSM API calls. Because SSM is part of the broader Systems Manager portfolio, it is easy to dramatically over-permission access if you’re not careful. So, we will break down how to configure the server and client side securely.

Configuring SSM on EC2 Instances

Any EC2 instance will need to have the aws-ssm-agent package installed. Most modern CentOS/RedHat, Ubuntu and Amazon distributions include this automatically. In addition, the IAM permissions for the instance need to allow SSM access. Make sure your IAM instance profile includes something like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": "*"
    }
  ]
}

Note that if you’re adding or modifying an IAM instance profile on a running instance, you’ll either need to reboot it or be patient for about 10 minutes to see the new permissions take effect. Note that the IAM Managed Policy AmazonSSMManagedInstanceCore gives comparable access and can be used.

Configuring SSM Access for Users

Users in the PowerUsers or AdministratorAccess groups have SSM access natively. Outside of that, it is necessary to grant SSM access explicitly. This is a minimum policy to grant access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": [
                "arn:aws:ec2:[REGION]:[ACCOUNT_ID]:instance/instance-id",
                "arn:aws:ssm:[REGION]:[ACCOUNT_ID]:document/SSM-SessionManagerRunShell" 
            ],
            "Condition": {
                "BoolIfExists": {
                    "ssm:SessionDocumentAccessCheck": "true" 
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeSessions",
                "ssm:GetConnectionStatus",
                "ssm:DescribeInstanceProperties",
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:TerminateSession",
                "ssm:ResumeSession"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:session/${aws:userid}-*"
            ]
        }
    ]
}

Configuring Your Local Environment

You’ll need the AWS SSM Manager Plugin for the best experience. Installing via brew is easy enough:

brew tap aws/tap
brew tap dkanejs/aws-session-manager-plugin
brew install awscli aws-session-manager-plugin

On Windows, you can download the plugin from AWS.

Connecting to an EC2 Instance Using SSM

Once the agent is installed and permissions are configured, connecting is easy:

aws ssm start-session --target [INSTANCE_ID]

Of course, this requires that you know the instance id you are connecting to. We’ll get to that momentarily.

Port Forwarding

Port forwarding with SSM allows secure connections to services on your instance. Here’s a snippet for forwarding a local port to a port on your instance:

aws ssm start-session --target [INSTANCE_ID] \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["80"],"localPortNumber":["8080"]}'

This command forwards local port 8080 to port 80 on the EC2 instance. Replace instance-id with your instance’s ID. Then connect to https://localhost:8080/ on your local to connect to port 80 on your EC2 instance. You can also connect to services like internal URLs and databases on private subnets using the StartPortForwardingSessionToRemoteHost command. For example, to connect to a database through a relay bastion, you can use remote port forwarding:

aws ssm start-session --target [INSTANCE_ID] \
--document-name AWS-StartPortForwardingSessionToRemoteHost \
--parameters '{"host":["pgdb.cqutuw8xa8zf.us-east-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["5432"]}'

Once the connection is established, you’ll be able to connect to your postgres database on localhost. If you’re connecting with TLS, you can use a hosts file entry and connect over the URL to avoid certificate validation errors. Edit your hosts file (/etc/hosts on Linux/macOS, c:\windows\system32\drivers\etc\hosts.txt) and add a line:

127.0.0.1 pgdb.cqutuw8xa8zf.us-east-1.rds.amazonaws.com

You can do the same to connect to internal admin UIs. The following would connect you to Kibana on a private OpenSearch cluster:

sudo aws ssm start-session --target [INSTANCE_ID] \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["vpc-logging-n21tg1kz9rty1crt3sd7k43uwnc.aos.us-east-1.on.aws"],"portNumber":["443"],"localPortNumber":["443"]}'

Note that we use sudo to be able to bind to port 443, since it is a secure port. Then, add it to your hosts file:

127.0.0.1 vpc-logging-n21tg1kz9rty1crt3sd7k43uwnc.aos.us-east-1.on.aws

Now connect to https://vpc-logging-n21tg1kz9rty1crt3sd7k43uwnc.aos.us-east-1.on.aws/_dashboards from your browser.

Making It Easier

Needing to know the instance ID is kind of a pain. There are a few ways to improve the situation. One is the ssm-utils wrapper written by Michael Ludwig. He wrote a great blog post on it here, but we’ll give a few key commands here.

Installing with PIP is easy:

sudo pip3 install aws-ssm-utils

See the instances checked in with AWS System Manager:

$ ssm-session --list
i-0bfa18a555c2dd124    ip-10-0-33-56.ec2.internal   web1        10.0.33.56
i-04256e7582f9119c6    ip-10-0-8-6.ec2.internal     backend1    10.0.8.6

Now I can connect using instance ID, IP address or name:

$ ssm-session 10.0.33.56
Starting session with SessionId: me-0e69d019bda4eac36
$

$ ssm-session backend1
Starting session with SessionId: me-039f74edd75a4adab
$

ssm-utils also supports SSH native tunneling with the ec2-ssh command. This means you can use conventional tools like rsync and scp:

$ rsync -e ec2-ssh file.txt ec2-user@backend1:file.txt

SOCKS Proxy

Perhaps the easiest way to connect to internal resources is the SOCKS proxy, which most browsers and database clients support. Using a SOCKS proxy gives VPN-like capability but without having to put a VPN endpoint on the Internet. The SSH session tunnels via SSM using your IAM credentials and your browser simply accesses internal resources natively.

To start a proxy, simply connect to any host inside your environment that has access to the resources you want to connect to. This is often a bastion on a private subnet made specifically for this purpose:

$ ec2-ssh ec2-user@bastion -N -D 8080
INFO: Resolved instance name 'bastion' to 'i-09256e7f9e4d119d3'
INFO: Using SSH key from ~/.ssh/id_rsa.pub - should work in most cases

This will create the proxy but not an interactive shell. Once established, you can update your browser settings to use the proxy server (127.0.0.1:8080). You can now access arbitrary URLs in private subnets as if you had a VPN connection. Database clients like dBeaver support SOCKS proxies as well. This method is much better than port forwarding, as you can use encryption with certificate verification with the proxy.

By default, all of your web browsing will route through the proxy, including things like Google or the AWS console. This is not a big deal if you are going to use this sporadically, but for daily usage, FoxyProxy and Firefox Containers are strongly recommended. You can then assign a container group to your proxy so that your normal browsing is unaffected.