The impacts of using custom networking with the AWS VPC CNI

Jeremy Cowan
3 min readAug 29, 2019

--

This post was inspired by https://github.com/aws/amazon-vpc-cni-k8s/issues/331.

Introduction

The AWS VPC CNI has a feature known as custom networking that allows you assign IP addresses to pods from a secondary VPC CIDR range. Ordinarily, pods are assigned an IP addresses from the host’s subnet, but this can be problematic if there aren’t enough IP addresses in the host’s subnet CIDR range to assign IPs to all your pods. Custom networking overcomes this issue by assigning pods IP addresses from a secondary CIDR that can be created from the 100.64.0.0/10 and 198.19.0.0/16 ranges. See the what’s new announcement, Amazon EKS now supports additional VPC CIDR blocks, for further information.

Custom AMIs and max-pods

Shortly after EKS was launched, AWS created a GitHub project called amazon-eks-ami. Within the repository are a collection of Packer scripts for customers to create their own custom AMIs. You’ll also find a file called eni-max-pods.txt which is used to set the value of --max-pods on the kubelet. The value controls the maximum number of pods that can be scheduled onto a particular instance type. For example, a c5.large instance can support a maximum of 29 pods.

The formula for calculating max-pods is as follows:

(number of ENIs*(number of secondary IP addresses-1))+2

The number of ENIs is the maximum number of ENIs that a particular instance can support. The number of secondary IP addresses is the maximum number of secondary IP addresses that can be assigned to each ENI. We subtract 1 from total number of secondary IP addresses because 1 IP address is reserved for each ENI that is attached to the instance.

AWS published a table that shows the maximum number of ENIs and secondary IPs per instance type.

We add 2 to the total value because the kube-proxy and aws-node pods are run with hostNetwork and therefore don’t consume an IP.

When you use custom networking, the formula for max-pods changes because pods are no longer scheduled on the host’s primary ENI unless they’re configured to use hostNetwork. With custom networking the formula becomes:

((number of ENIs-1)*(number of secondary IP addresses-1))+2

Let’s have a look at an example. Say you use a c5.large for your worker nodes. A c5.large can support a maximum of 3 ENIs and 10 secondary IP addresses per ENI. This means that a c5.large with custom networking can support a maximum of ((3-1)*(10-1))+2 or 20 pods. This is 9 fewer pods that a c5.large can support without custom networking. Pod density is lower with custom networking because the primary ENI is reserved.

The new formula works so long as you don’t run additional pods with hostNetwork. When pods are run with hostNetwork they don’t consume a secondary IP but they still count towards max-pods. Now suppose we run 5 pods with hostNetwork on a c5.large with custom networking. With max-pods set to 20 we can only schedule another 15 pods onto that instance, however, as none of the 5 pods are consuming a secondary IP, we should really be able to schedule another 20 pods onto that instance. While this is probably an edge case, it’s good to be aware of how max-pods and custom networking affect the number of pods you can run on a particular instance.

If you are going to configure the VPC CNI to use custom networking and you’re building AMIs with the aforementioned Packer scripts, replace the eni-max-pods.txt file with your own file using the updated formula to calculate max-pods.

Update

Also, if you are already using custom networking and your cluster is impacted by CVE-2019–9512 and CVE-2019–9514, you need to upgrade the CNI to v1.5.3 immediately!

Conclusion

Custom networking is especially useful in IP constrained environments where there aren’t enough IP addresses available in the primary CIDR to assign IP addresses to pods. It will, however, impact your pod densities and the max-pods setting on the Kubelet. Consider these impacts before you use custom networking.

--

--

Jeremy Cowan
Jeremy Cowan

Written by Jeremy Cowan

Jeremy Cowan is a Principal Container Specialist at AWS

Responses (1)