VM not accessible, you might also need to check datastore for space

Last weekend I got a support call from database team to check, why they lost connectivity to a database server VM. Firstly I tried to ping  this VM but it was not reachable so have to login to vCenter to see what’s wrong with that VM. First thing that I noted, VM was powered on but had a message sign on it and when tried to open VM console for further investigation, got a datastore space related popup question,

Clicked on retry but the pop-up came up again.

As this popup question suggests, there was not enough space in datastore for the VM to breath and this is because the VM was running on Snapshot (******-000001.vmdk).

Even you would see the same question in VM summary page.

When I checked the datastores where the VM disks reside, I was amazed to see this,

Here one may ask why this happened, do you not having Storage Cluster/SDRS and the answer is no we don’t (that’s a different story). Thin provisioning is also not the case here but the Snapshot of this large VM is (this VM having one or more TB disk attached). This snapshot was created by VM backup tool during backup but at the same time there was some activity going on the server so it grows unexpectedly, eaten all available datastore space and cause this issue.

(if you are thinking why the hell we are taking image level backup of database VM drives, please don’t bother to ask me as I also couldn’t find the logic of that)

So to fix this what we need to do is, check all the datastores where VM disks reside (in VM summary you would see the datastore in question with space error/warning alert), create space for VM to breath and once you are done, go to VM summary,

Select Retry option in VM question and click ok.

VM should be accessible now (now you might also take a look at backup server to see if backup was completed, if so but the snapshot is still there then delete the snapshot and if the backup job is still in running state then no worries snapshot would automatically delete once the backup complete).

Note: You may also see an open VM console MKS error during open VM console (like /vmx file not accessible or unable to open) due to space crunch in datastore.

That's it... :)