Open VM Console "Unable to connect to the MKS: ......" errors

You may have seen one of this kind of MKS errors while trying to open VM console, 

                

                

First of all you may be wondering what the ‘MKS’ part of the error message stands for, well you’ll be disappointed to know that it isn’t an acronym for something high tech and very complicated but rather is stands for; mouse, keyboard, screen and the error is your vSphere installed system simply unable to map this to the guest OS, hence the open VM console failed.

Most of the time "Unable to connect to the MKS:” errors are network issues (DNS/firewall, port 902 issue). Either your system (where the vSphere client is installed) is not able to resolve host/VM name or port 902 is not open.

You can check the name resolution easily by using ping command, just open the CMD and try to ping the esxi host/VM and see if the name resolution is happening.

If system is unable to resolve the name, check your network setup/firewall config. This error is caused by your vSphere client not able to communicate with the ESXi host directly.

After verifying DNS, open a command prompt on the vSphere Client machine and run these commands:
ipconfig /flushdns
ipconfig /registerdns

If name resolution is not happening, as a workaround of dns issue you may edit the C:\Windows\System32\drivers\etc\hosts file and add esxi host's dns entry there. 

If it’s not DNS then:

Before doing anything else first vMotion the affected VM to any other and see if the issue still persist.

Verify that the ESXi/ESX host and the workstation running the vSphere Client are correctly synced to an NTP service. This is required to satisfy SSL handshaking between the vSphere Client or and the ESXi/ESX host.

A restart of the esxi host management agents would also fix any of the below error.

 

In case you are using Esxi 6.0.x and getting following error “Unable to open MKS: Internal Error” while opening VM console for any VM then as listed in kb# 2116542,

This issue is caused by the SSL certificates are updated by the ESXi host when connecting to the new vCenter Server.  These certificates are used by the console and may not be updated with the virtual machine running.

This is a known issue affecting ESXi 6.0.

To resolve this issue, use one of these options:

• Power off and Power on the virtual machine.

• Migrate the virtual machine to another ESXi host using vMotion.

• Suspend and then Resume the virtual machine.

Note: The virtual machine must be powered off for the changes to take effect.  A warm reboot will not resolve the issue.

In earlier versions of Esxi you may also try this for above error, 

Select affected Esxi Host => Configuration > Advanced Setting

Now Go to Config => Security => host

Uncheck the Config.Defaults.security.host.ruissl

In case you are using HP ProLiant server and getting open VM console errors like,

Unable to contact the MKS: Could not connect to pipe\\.\pipe\vmware-authdpipe
                                                    or
 Unable to connect to the MKS: connection terminated by server.

                                                    or

or

 It could be related to hp-ams related bug, if above didn't work then check if a latest version of hp-ams is available and update the same if available using below command.

esxcli software vib install -d /datastore/directory/hp-ams-esxi5.5-bundle-10.0.1-2.zip 

Or as a workaround for the time being you may stop the hp-ams process,

/etc/init.d/hp-ams.sh stop 

or Uninstall the hp-ams by using the below command,

esxcli software vib remove -n hp-ams

Note: for any of the above open VM console error, not necessarily all VM running on the host would be affected by this issue. 
If anyone came across any of the MKS error not listed here please let me know in comment area.

Reference: kb# 749640.

That's it... :)