Azure backup | Staging storage account not visible during restore

Recently, I got this query from one of the team members working on a VM restore task using Azure Backup encountered an issue. He was unable to find the available storage accounts for staging location selection despite there being a storage account in the subscription.

I had come across this issue earlier and was aware about the cause however couldn't recall at that moment so spent some time looking into the documentation and when found thought of making a note as who knows about the next time😉or let's hope Microsoft will remove this limitation😊

You can see the same behavior in the following related screenshots,

As you can see, I have three storage accounts in my test subscription, where one is in another region, and out of the other two one is having Standard_ZRS SKU.

During restore, as you can see the storage account having ZRS sku is not available for selection.

BTW if you click on the info icon in front of the "Staging Location" option, it will show you the related hint as follows😄

Reference:  Azure Backup Restore documentation.

That's all for today, thanks...

Azure AKS cluster upgrade error "... vmss has reached its limit of 10 models ..."

In this short post, we will discuss a terminal provisioning state failed error encountered during the AKS cluster upgrade. The control plane was successfully upgraded, but the AKS node pool upgrade subsequently failed with the following error.

Here, as we can see from the error, it pertains to the vm scale set model, so lets first discuss it. Essentially, the VMSS model represents the desired state of the vmss as a whole and implies a property of the scale set which affects VMs, for example the VM size, the OS version or an extension.

For AKS node pools the orchestration mode is set to Uniform (designed to be a collection of similarly configured virtual machines) and the VMSS will have the default model upgrade policy, set to manual, where VMs can have different models but that is restricted to 10 models overall (all the VMs in the VMSS can have 10 unique configurations).

So, you might see this error if you are having AKS node pool with 10+ nodes and they are not having the latest model. You can check this by going to the respective node pool VM Scale Set =>Instances, check the Latest Model column and it would show you Yes/No.

To proceed from here, select the required instances showing the "Latest Model as NO" and click on "Upgrade" from the available highlighted options on top.

As this might reboot the instance and impact the availability so make sure that you are doing it caution.

Once you have upgraded the instance to the latest model then re-initiate the AKS upgrade again and this time you should not see this error.

If you're interested in learning more about the scale set upgrade policies then can check the related documentation available here, 

VM Scale Set model

Upgrade policies for Virtual Machine Scale Sets

I hope you find this information useful. Thank you!

Azure AppGW http error 403 and outdated browser

This post is about encountering an AppGW HTTP error 403 while attempting to access a site published through Azure Application Gateway v2 with WAF enabled,

Upon reviewing the AppGW logs (category== ApplicationGatewayFirewallLogs), noticed the error was specific to a single client IP and in error msg related detail there was mention of browser cookies. looking at this recommended the user to try accessing the site using a different browser (previously Google Chrome) and the site was accessible.

Following that, we updated Google Chrome to the latest version and rechecked and this time, we encountered no errors, and the site was accessible without any issues.

The main takeaway here is that outdated browsers can sometimes trigger the web application firewall to block incoming requests. Therefore, if you come across the related indication in the AppGW logs then check the url using a different browser also the version of the browser in question and update it to the latest.

Related sample kql query to use for AppGW logs,

AzureDiagnostics

| where TimeGenarated > ago(1h)

| where Category== "ApplicationGatewayFirewallLog"

| where clientIp_s== "<required sourceIP" and requestUri_s contails "/path in your case"

In my case, extract from query output,

Message: Detects MySQL comment-/space-obfuscated injections and backtick termination

OWASP CRS ruleSetVersion_s: 3.2

ruleGroup_s: REQUEST-942-APPLICATION-ATTACK-SQLI

details_message_s: Pattern match (?i:(?:(?:(?:(?:trunc|cre|upd)at|renam)e|(?:inser|selec)t|de(?:lete|sc)|alter|load)\s*?\(\s*?space\s*?\(|,.*?[)\da-f"'`]["'`](?:["'`].*?["'`]|(?:\r?\n)?\z|[^"'`]+)|\Wselect.+\W*?from)) at REQUEST_COOKIES.

details_data_s: {,"campaigns":{"34645675687werwe4567rit6":{ found within [REQUEST_COOKIES:ORA_PERS:{"ids":["-23434645757657"],"campaigns":{"":{"activeBlocks":["c1","C2","C3","C4"],"pointer":"E1","event":"-687897890978860392"}}}]}

If you're interested in learning more about HTTP error codes, you can explore the following links:

HTTP response status codes

HTTP response codes in Application Gateway

I hope you find this information useful. Thank you!

VMware vExpert 2022

I am very honored to be named a VMware vExpert again... and yeah it was announced on Feb 17, 2022 however at that time somehow missed that email ;)

I would like to Congratulations to all those who made in the vExpert 2022 list.

VMware vExpert directory is available here.. https://vexpert.vmware.com/directory

Thank you, VMware... :)

AWS Single Sign-on and Azure AD Application Certification rotation

In this quick post would discuss the process and steps involved in rotating the expiring Azure AD application certification configured for AWS SSO login. 

This is applicable where you have your AWS account SSO configured with Azure Active Directory and the associated application password is about to expire or maybe already expired.

Before you start, make sure to have the appropriate AWS IAM and Azure AD permission or involve the teams having the required access to create an application certificate (in Azure) and rotate the same in AWS.

Now, login to AWS and take the backup of currently used metadata.

1. Login to AWS => Go to IAM => Click on Dashboard or from the IAM menu, click on Identity Provider
2. Click on Azure AD => From Metadata Document section, Download the current metadata file for backup purpose

Now Login to Azure,

1. Go to Azure Active Directory => Select Enterprise applications from left menu options
2. From the Enterprise applications section, Select the correct AWS Application used for SSO 
3. No on the AWS Application screen, go to Single Sign-on option => SAML Signing Certificate and click Edit
   
4. On SAML Signing Certificate Page, Create a new Certificate, Save and mark it as Active, close the window
   
5. Now on SAML Signing Certificate Page, verify the certificate Expiry date and Download the Federation Metadata XML
   
   
   
   
6. Go to AWS account IAM Identity Provider Section, Steps are mentioned above
7. Within the Metadata Document section, this time Click on Replace Metadata, on pop-up window Type replace and Click on Replace tab. Just in case if you didn't download the current metadata file earlier, do that so just in case of any issue you could revert
   
8. Now browse and select the Federation Metadata XML file downloaded after Azure AD application certificate rotation and click open
9. It would take the next few seconds and you are done.
10. Test your AWS Single Sign-on URL, you can also perform the testing from within the Azure Application SAML bases Single sign-on page.

Note: If you are using an AD account to replace the AWS Identity provider Metadata then make sure to log in prior to marking the newly created Azure application certificate active. Also, don't refresh the AWS login page until you replace the metadata.

To avoid this, simply use your AWS root account ;)

Related Demo: 

That's it, thanks :)