Using AWS SSM to manage private bastions

Stop using SSH for bastions!

In this article, we will see what a bastion is, why SSH connection to EC2 is not recommended, and an alternative solution provided by AWS Session Manager service. 

First of all, what is a bastion?

Usually, we locate sensitive resources like databases in a private subnet to hide them from the internet, improve security, and avoid the possibility of DDoS. Therefore, we need to think about how to access the database when no direct internet connection is available. 

One of the ways to access a private subnet is to spin up a little instance in a public subnet in the same network. This instance is like a bridge: first, we connect to it, and from there, we can connect to the database.

This is what we call a bastion — a server used to provide access to a private network from an external network.

Another way to access a private subnet is to create a VPN connection between the database’s VPC and the user’s network. However, there is a place for bastions even in the VPN world. For instance, they can provide an additional security layer when the database is accessible only internally or from bastion’s security group. This schema is usually used in production databases, to control the access for database maintenance and to cover security compliance requirements.

This is the reason why the bastion host is called so — like a citadel, where the princess hides to stay safe.

Other use-cases for a bastion can be, for example: allowing temporary, guest, and applicative access to the database when we don’t want to or cannot use a VPN.

What’s the problem with using bastions this way?

Bastions are located in public subnets and are accessible from the internet. This means that bastion hosts are exposed to potential attacks, and they must be extra secured:

• We need to manage security groups to define which IPs are allowed to access the bastion (or worse, if we don’t manage rules, we need to allow access from anywhere)
• Patching can be an issue; the bastion server must be updated regularly. 
• We usually allow only SSH ingress connections. However, the biggest issue is SSH key management. Who is permitted to upload the keys? How can it be done securely? Even if key issuing is automated and managed (for example by Okta), what about keys rotation?

Let’s look at two real-life examples I faced where using SSH keys was challenging:

• Company A managed one bastion for “authorized personnel” (like team leaders), with a cluster of trusted keys. When they ran a script, it created a user for a team member on a second bastion and copied their public key into that user from the public keys present in the existing pool on the first bastion… Well, it works and is more or less secure, but managing the keys this way is a complicated and exhausting mission. 
• In Company B, one public key was uploaded into the instance, and a private key was shared between the developers. No comment.

AWS Session Manager

Fortunately, AWS Cloud gives us all the functionality to solve the above issues. The provided tool is called AWS Systems Manager Session Manager. It’s a powerful connection tool for EC2, and it has a lot of useful features, and the main ones we are interested in are:

• SSM allows us to connect to an EC2 instance located inside a private subnet,
• … with no inbound security group rules, 
• … with no ssh keys, 
• … and with no bastions.
• The access is given by time-limited sessions. 
• All the permissions are flexible and managed through IAM Roles.

In the same way, if we want to manage the bastion host as a tunnel for a database, SSM allows us to create it inside the private subnet and restrict the access for tunneling (port-forwarding) only.

Would you like to know how to set up a private bastion with an SSM connection, launch the DB tunnel on it, and use IAM roles for accessing the tunnel? Read the Terraform guide.