Azure AKS cluster upgrade error "... vmss has reached its limit of 10 models ..."

In this short post, we will discuss a terminal provisioning state failed error encountered during the AKS cluster upgrade. The control plane was successfully upgraded, but the AKS node pool upgrade subsequently failed with the following error.

Here, as we can see from the error, it pertains to the vm scale set model, so lets first discuss it. Essentially, the VMSS model represents the desired state of the vmss as a whole and implies a property of the scale set which affects VMs, for example the VM size, the OS version or an extension.

For AKS node pools the orchestration mode is set to Uniform (designed to be a collection of similarly configured virtual machines) and the VMSS will have the default model upgrade policy, set to manual, where VMs can have different models but that is restricted to 10 models overall (all the VMs in the VMSS can have 10 unique configurations).

So, you might see this error if you are having AKS node pool with 10+ nodes and they are not having the latest model. You can check this by going to the respective node pool VM Scale Set =>Instances, check the Latest Model column and it would show you Yes/No.

To proceed from here, select the required instances showing the "Latest Model as NO" and click on "Upgrade" from the available highlighted options on top.

As this might reboot the instance and impact the availability so make sure that you are doing it caution.

Once you have upgraded the instance to the latest model then re-initiate the AKS upgrade again and this time you should not see this error.

If you're interested in learning more about the scale set upgrade policies then can check the related documentation available here, 

VM Scale Set model

Upgrade policies for Virtual Machine Scale Sets

I hope you find this information useful. Thank you!