Introducing the aws-ssm-agent-installer

When managed node groups was first introduced, it automated the provisioning of Kubernetes worker nodes and orchestrated the replacement of those nodes when a new version of the EKS optimized AMI was released. Months later, AWS added the ability to use your own custom AMI with managed node groups and customize the launch template used by managed nodes groups, including EC2 user data. This allowed customers to use a different flavor of Linux for their worker nodes and/or add customizations to the EKS Optimized AMI through EC2 user data. The downside was that customer who chose to customize the EKS optimized AMI would be responsible for maintaining that AMI going forward, i.e. they would have to rebuild their custom AMI each time AWS released a new version of the EKS Optimized AMI.

aws-ssm-agent-installer to the rescue

The aws-ssm-agent-installer is an Open Source project that allows you to use a Kubernetes DaemonSet to install the SSM agent onto worker nodes as they join the cluster. This allows you to use the stock EKS Optimized AMI with managed node groups instead of maintaining your own AMI. If additional customizations are needed, they can be installed and managed through AWS Systems Manager. You’ll also have access to the full breath of features that are included with System Manager such as:

• Viewing patch compliance details
• Running automation runbooks to resolve issues
• Deploying further customizations
• Inventorying the software running on each node
• Using session manager to start an interactive shell on the worker nodes without having to manage SSH keys or bastion hosts
• Getting real-time performance metrics for CPU, memory, network traffic, and disk IO

It’s also GitOps friendly in that you can add the manifest for the DaemonSet to a Git repository that the cluster uses to configure itself.

Why not add the SSM agent to the EKS Optimized AMI?

Ideally the SSM agent would be bundled with the EKS Optimized AMI. Until that happens, you need to install it yourself. The aws-ssm-agent-install gives you a method for installing the SSM agent onto each node in your cluster without having to modifying the Optimized AMI or the node group’s launch template. It is also versatile in that you can use it to install other configuration agents like Chef simply by modifying the DaemonSet’s ConfigMap.

Minimizing the attack surface

While the DaemonSet’s init container runs as a privileged container, it is replaced with a pause container with minimal privileges once the init container exits. Since the pause container has a minimal attack surface and consumes very few resources you can continue running it in your cluster after the agent has been installed. Allowing it to run continually will guarantee that the SSM agent will be installed onto all new workers as they join the cluster.

Using KMS with the SSM Agent

For an added layer of security you can use KMS to encrypt Session Manager sessions. This will encrypt all session data between EC2 and the user’s machine. This is in addition to the TLS 1.2 encryption that AWS already provides by default. For further information about this option, see https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-enable-encryption.html.

If you intend to monitor instance performance through Fleet Manager, you have to enable KMS session encryption.

Worker nodes must be given permissions to the KMS key used to encrypt communications between the worker nodes and SSM. Below is a sample IAM policy that can be added to the EKS nodes EC2 instance role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:<REGION_ID>:<ACCOUNT_ID>:key/<KMS_KEY_ID>"
        }
    ]
}

This IAM policy can be added to the worker nodes a variety of ways. Below is an example of using an eksctl config file to attach the policy to the EKS node instance role, in a managedNodeGroup element.

managedNodeGroups:
  - name: mng-1
...
    iam:
      attachPolicyARNs:
...
        - arn:aws:iam::207726343182:policy/KmsKeyUserSsmOps

Conclusion

The aws-ssm-agent-installer give you a mechanism to install the SSM agent onto your worker nodes without having to modify the stock EKS Optimized AMI. Furthermore, once the agent is installed onto the node, you can use SSM’s breath of features to manage and troubleshoot it.

I welcome your feedback. If you have a suggestion or an issue, please open a PR or an issue in the GitHub project.