Running Weave Net on EKS

Background

The AWS VPC CNI plug-in is installed by default when you provision an EKS cluster. Unlike an overlay network, the VPC CNI plug-in assigns routable IP addresses from your VPC to Kubernetes pods. This routable IP allows you to use VPC flow logs to capture source IP/port, destination IP/port and protocol from traffic emanating from and sent to pods. All of this information can then be fed to GuardDuty, a service which continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts and workloads. Nevertheless, there are times when using the VPC CNI plug-in is not viable choice, e.g. when there aren’t enough IP addresses or CIDR blocks available.

Why Weave Net

Weave Net is a popular CNI plug-in for Kubernetes because it is feature-rich and relatively easy to install. With Weave Net, there’s no need to build or maintain a separate cluster store for storing the network configuration. Instead, nodes exchange topology messages with each other using the gossip protocol. Weave Net also has native support for multicast traffic as well as Kubernetes network policies.

Preparing your environment

While it’s possible to reconfigure existing nodes to use Weave Net, https://github.com/weaveworks/weave/issues/3335#issuecomment-441522517, it’s easier to start with a nodeless cluster. For simplicity’s sake, I recommend using eksctl.

eksctl create cluster --name braavos --nodes 0 --ssh-access --region us-east-1

Executing the above command will:

• Create a dedicated VPC
• Provision the Kubernetes control plane and update your kubeconfig file to use the new cluster
• Create an autoscaling group with a launch configuration for worker nodes
• Create a pair of security groups for the worker nodes and the Kubernetes control plane

Once the control plane has been provisioned, delete the aws-node daemonset. aws-node is the daemonset for the AWS VPC CNI plug-in and is installed by default when you create an EKS cluster.

kubectl delete ds aws-node -n kube-system

Next, open the EC2 console and find the security group that the resembles the name eksctl-<clustername>-cluster-ClusterSharedNodeSecurityGroup. Add the following rules to the security group, if necessary:

The source field should reference the security group ID of the node group. You must permit traffic to flow through TCP 6783 and UDP 6783/6784, as these are Weave’s control and data ports.

Installing Weave Net

Of the 3 CNI plug-ins I’ve written about so far, Weave Net is the easiest to install and configure because it does not require a cluster store like etcd or zookeeper.

To install the Weave Net CNI simply execute the following command:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Once the Weave Net daemonset has been deployed, update the desired and maximum count parameters on the autoscaling group for your worker nodes. This will add worker nodes to your cluster and automatically join them to the Weave network.