3. Cluster Networking | EKS Anywhere

Publish date: 2024-05-06

EKS Anywhere cluster networking

Cluster Networking

EKS Anywhere clusters use the clusterNetwork field in the cluster spec to allocate pod and service IPs. Once the cluster is created, the pods.cidrBlocks, services.cidrBlocks and nodes.cidrMaskSize fields are immutable. As a result, extra care should be taken to ensure that there are sufficient IPs and IP blocks available when provisioning large clusters.

apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: Cluster metadata:  name: my-cluster-name spec:  clusterNetwork:  pods:  cidrBlocks:  - 192.168.0.0/16  services:  cidrBlocks:  - 10.96.0.0/12 

The cluster pods.cidrBlocks is subdivided between nodes with a default block of size /24 per node, which can also be configured via the nodes.cidrMaskSize field. This node CIDR block is then used to assign pod IPs on the node.

Warning

The maximum number of nodes will be limited to the number of subnets of size /24 (or nodes.cidrMaskSize if configured) that can fit in the cluster pods.cidrBlocks.

The maximum number of pods per node is also limited by the size of the node CIDR block. For example with the default /24 node CIDR mask size, there are a maximum of 256 IPs available for pods. Kubernetes recommends no more than 110 pods per node.

Ports and Protocols

EKS Anywhere requires that various ports on control plane and worker nodes be open. Some Kubernetes-specific ports need open access only from other Kubernetes nodes, while others are exposed externally. Beyond Kubernetes ports, someone managing an EKS Anywhere cluster must also have external access to ports on the underlying EKS Anywhere provider (such as VMware) and to external tooling (such as Jenkins).

If you are responsible for network firewall rules between nodes on your EKS Anywhere clusters, the following tables describe both Kubernetes and EKS Anywhere-specific ports you should be aware of.

Kubernetes control plane

The following table represents the ports published by the Kubernetes project that must be accessible on any Kubernetes control plane.

ProtocolDirectionPort RangePurposeUsed By
TCPInbound6443Kubernetes API serverAll
TCPInbound10250Kubelet APISelf, Control plane
TCPInbound10259kube-schedulerSelf
TCPInbound10257kube-controller-managerSelf

Although etcd ports are included in control plane section, you can also host your own etcd cluster externally or on custom ports.

ProtocolDirectionPort RangePurposeUsed By
TCPInbound2379-2380etcd server client APIkube-apiserver, etcd

Use the following to access the SSH service on the control plane and etcd nodes:

ProtocolDirectionPort RangePurposeUsed By
TCPInbound22SSHD serverSSH clients
Kubernetes worker nodes

The following table represents the ports published by the Kubernetes project that must be accessible from worker nodes.

ProtocolDirectionPort RangePurposeUsed By
TCPInbound10250Kubelet APISelf, Control plane
TCPInbound30000-32767NodePort ServicesAll

The API server port that is sometimes switched to 443. Alternatively, the default port is kept as is and API server is put behind a load balancer that listens on 443 and routes the requests to API server on the default port.

Use the following to access the SSH service on the worker nodes:

ProtocolDirectionPort RangePurposeUsed By
TCPInbound22SSHD serverSSH clients
Bare Metal provider

On the Admin machine for a Bare Metal provider, the following ports need to be accessible to all the nodes in the cluster, from the same level 2 network, for initially network booting:

ProtocolDirectionPort RangePurposeUsed By
UDPInbound67Boots DHCPAll nodes, for network boot
UDPInbound69Boots TFTPAll nodes, for network boot
UDPInbound514Boots SyslogAll nodes, for provisioning logs
TCPInbound80Boots HTTPAll nodes, for network boot
TCPInbound42113Tink-server gRPCAll nodes, talk to Tinkerbell
TCPInbound50061Hegel HTTPAll nodes, talk to Tinkerbell
TCPOutbound623Rufio IPMIAll nodes, out-of-band power and next boot (optional )
TCPOutbound80,443Rufio RedfishAll nodes, out-of-band power and next boot (optional )
VMware provider

The following table displays ports that need to be accessible from the VMware provider running EKS Anywhere:

ProtocolDirectionPort RangePurposeUsed By
TCPInbound443vCenter ServervCenter API endpoint
TCPInbound6443Kubernetes API serverKubernetes API endpoint
TCPInbound2379ManagerEtcd API endpoint
TCPInbound2380ManagerEtcd API endpoint
Nutanix provider

The following table displays ports that need to be accessible from the Nutanix provider running EKS Anywhere:

ProtocolDirectionPort RangePurposeUsed By
TCPInbound9440Prism Central ServerPrism Central API endpoint
TCPInbound6443Kubernetes API serverKubernetes API endpoint
TCPInbound2379ManagerEtcd API endpoint
TCPInbound2380ManagerEtcd API endpoint
Snow provider

In addition to the Ports Required to Use AWS Services on an AWS Snowball Edge Device , the following table displays ports that need to be accessible from the Snow provider running EKS Anywhere:

ProtocolDirectionPort RangePurposeUsed By
TCPInbound9092Device ControllerEKS Anywhere and CAPAS controller
TCPInbound8242EC2 HTTPS endpointEKS Anywhere and CAPAS controller
TCPInbound6443Kubernetes API serverKubernetes API endpoint
TCPInbound2379ManagerEtcd API endpoint
TCPInbound2380ManagerEtcd API endpoint
Control plane management tools

A variety of control plane management tools are available to use with EKS Anywhere. One example is Jenkins.

ProtocolDirectionPort RangePurposeUsed By
TCPInbound8080Jenkins ServerHTTP Jenkins endpoint
TCPInbound8443Jenkins ServerHTTPS Jenkins endpoint

ncG1vNJzZmiZnq7EqbHRnmWeo6Njrq6t2ailmq%2BjY7CwuY6dppyrX5yytcDIp55mq6SWv7Wxw2inqKqkqHw%3D