Pod Networking

How do pods communicate with each other?

Kubernetes expects you to implement a networking solution on your own.

Kubernetes expectations

• Every pod should have an IP address
• Every pod should be able to communicate with every other pod on the same node
• Every pod should be able to communicate with every other pod on other nodes without NAT

Node 1 eth0: 192.168.1.11 v-net-0: 10.244.1.1/24

Node 2 eth0: 192.168.1.12 v-net-0: 10.244.2.2/24

Node 3 eth0: 192.168.1.13 v-net-0: 10.244.3.3/24

Creating a new node

In intro, we noticed the step sto attach to a namespace network

1. Create a bridge interface on the host, and mark it as up

   ip link add v-net-0 type bridge
   ip link set dev v-net-0 up
2. Assign the bridge interface an IP address. This becomes the gateway of our new virtual network. Each node gets its own address in a different subnet.

   ip addr add 10.244.{n}.1/24 dev v-net-0

This gives every node two unique IP addresses. The eth0 address, assigned by DHCP, and a v-net-0 address.

Routing between nodes

In a simple environment: each node will need routes to all the other internal node networking spaces:

# On Node 1
ip route add 10.244.2.0/24 via 192.168.1.12
ip route add 10.244.3.0/24 via 192.168.1.13

# On Node 2
ip route add 10.244.1.0/24 via 192.168.1.11
ip route add 10.244.3.0/24 via 192.168.1.13

# etc.

However, as this gets complex, it's easier to simply point all the nodes to a single default gateway/router. This would allow us to form a single large network 10.244.0.0/16 at the cluster level.

Creating a new container

In intro, we also see the steps to attach a container to a network namespace. This happens on every container create:

# Create the veth pair
ip link add ...

# Attach the pairs to the appropiate namespace
ip link set ...
ip link set ...

# Assign IP Addresses
ip -n {namespace} addr add

# Assign routes
ip -n {namespace} route add ...

# Bring up the interface
ip -n {namespace} link set ... up

How CNI Helps

Whenever kubelet starts up a container, it looks at its own:

• --cni-conf-dir=/etc/cni/net.d : To get the name of our networking script
• --cni-bin-dir=/etc/cni/bin : To get the directory where the script lives

Then calls ./net-script.sh add {container} {namespace} to setup the proper networking.

On delete, it will also call a ./net-script.sh del command accordingly.

The /opt/cni/bin direcetory has all of the plugins installed as executables:

• bridge
• dhcp
• portmap
• etc.

The /etc/cni/net.d directory has a set of config files. This is where Kubelet looks to find out what file to use.

* 10-bridge.conf # json file!

If there are multiple file here, it will use the first one in alaphabetical order.

{
   "cniVersion": "0.2.0",
   "name": "mynet",
   "type": "bridge",
   "bridge": "cni0", // name of interface
   "isGateway": true, // Should network get an IP address to act as a gateway?
   "ipMasq": true, // Should NAT rules be added for IP Masquerading
   "ipam": {
      "type": "host-local", // addresses are managed locally on this server
      "subnet": "10.22.0.0/16", // range of addresses assigned to pods on this host
      "routes": [
         { "dst":  "0.0.0.0/0" } // routes assigned to pods
      ]
      
   }
}

WeaveWorks CNI Plugin

Think of the Kubernetes cluster as our company, and the nodes as office sites. With each site we have different departments, and in each department we have differnt offices.

Networking agents are installed on each node, and are very well connected to understand connectivity. CNI will wrap the package to expedite networking. Weave creates its own bridge.

When packets are being sent between nodes, weave intercepts then encapsulates the packet and sends it along its way.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"