Designing the Azure Network
When creating the initial cluster, it was for development only and only one node was needed. For the interests of time, the simplest network CNI was used: kubenet.
Now that the production deployment is being done, it's time to step back and see what is best for the long term.
The VNet created by the kubenet CNI was 10.0.0.0/8 which addresses 10.0.0.1 -> 10.255.255.254. Basically consuming all network addresses in that address space.
However, the only subnet created was 10.240.0.0/16 which addresses 10.240.0.1 - 10.240.255.254.
Now it should be possible to reuse the VNet by creating other subnets in it, but having it named 'aks-vnet-41071798' implies to any future developer or devops person that this vnet is only for AKS. This is true, but AKS doesn't need that many addresses.
Furthermore, when setting up a Azure Database for Postgresql Flexible database, it also needs a subnet.
Additionally, virtual machines need to be included as well as there is a requirement to run PowerBi to talk to the db.
An additional consideration is that other resource groups may also have networking needs.
Now it is possible to create many networks, each using the same address space, but do not talk to each other, however if they do need to, they can be peered in order to do so.
To do this, the total address space, 10.0.0.0/8, was divided into 'subnets' of size 10.X.0.0/16. This makes 256 blocks of 65,536 addresses. They are addressed in this manner:
Block 1 - 10.0.0.0/16
Block 2 - 10.1.0.0/16
Block 3 - 10.2.0.0/16
...
Block 256 - 10.255.0.0/16
Then one or more of these blocks can be assigned to any VNet created. In this case an 'app-vnet' was created and it was assigned Block 2 and Block 3:
app-vnet: 10.1.0.0/16, 10.2.0.0/16
Block 1 will be for normal subnets. Block 2 will be for k8s service addresses as AKS needs a chunk of addresses that are not used in any subnet. They are added to the vnet to ensure they are part of the network, but not used in the subnets as is required. It would be nice if a subnet could be allocated for them then assigned. That is how our bare metal setup worked, but this is AKS so it's options are a little different.
In the app-vnet the address space can be broken down into 16 subnet blocks at: 10.X.0.0/20 each of which has 4,096 addresses.
These are addressed in this manner:
10.X.0.0
10.X.16.0
10.X.32.0
10.X.48.0
10.X.64.0
10.X.80.0
10.X.96.0
...
10.X.240.0
So I created 3 subnets:
- aks (for the aks pods) 10.1.0.0/20
- postgres (for the flexible postgres pods) 10.1.16.0/24
- vms (for any virtual machines needed)
I made the database subnet smaller so future databases can be placed in other smaller subnets, grouped near this one. Like 10.1.17.0/24**
The services have 65,536 addresses and the pods only have 4,096. This is a small cluster so the service addresses are quite a bit overkill and the pod addresses are likewise. It is not anticipated to ever reach close to either of these amounts. However, if that time comes, the application will be generating enough money that it will be financially feasible to rebuilt the cluster with more nodes and more address space.
Moving forward, each new VNet can continue to take one of the other 255 blocks of addresses. For example, the existing automatically created kubenet-vnet is using block 10.240.0.0/16. The others (10.0.0.0/16, 10.3.0.0/16, 10.4.0.0.0/16, ..., 10.239.0.0/16, 10.241.0.0/16, ..., 10.255.0.0/16) are still available.
None will conflict with each other, in the case that a peer needs to be created.
Now that this is done, there is some work to migrate the development database into the subnet. This is because when the db was created the 10.0.0.0/8 network address space was already allocated to the K8s Cluster. So when the database needed to be created, and it needs it's own subnet, it couldn't use that address space as it would conflict when the networks were peered. So it created one using the other RFC 1918 Network ranges: 172.16.0.0 - 172.31.255.255 (172.16/12 prefix). So then the networks were peered together and everything worked. Pods could communicate with the database. However, with the new design, things can be more organized in the address space and peering won't be necessary as they will be in the same vnet.
Reference Reading: