Remote connection failed with the error, outdated entry in the DNS cache

Last day I came across an issue where an user was not able to take remote of a windows VM using the server name however he was able to take remote of it using IP. He was getting an outdated DNS entry related error when taking RDP using server name, 

In first look it looks like a local dns cache related error so cleared the dns cache of his system by running ipconfig /flushdns and tried to connect again but end up with same error. Then I tried to connect to affected server from my system but as earlier got the login screen and after entering my credential as soon as I hit enter, end up with the same “…..This could be caused by an outdated entry in the DNS cache….” Error. Then I tried to login using the IP address of the server and was able to login without any issue.

After some investigation I noticed the affected system's time is different than the DC means not in sync with DC/DNS server so I restarted the Window Time Service and ask the user to try to login and this time user was able to login without any issue.

One can start the windows service either from services manager (Run => services.msc) or from command prompt. To restart the windows service from cmd run below commands.

net stop w32time

net start w32time 

Once the Windows time service would restart system  will sync the time with DC again and you will be able to login using server name.

Note: this issue is related to Windows time sync with DC so I'm sure it's not specific/limited to VM. 

That's It... :)

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem : HP ProLiant Gen9 Server

Last day when I logged in to vCenter I noticed one of the host with warning icon and upon checking in summary tab found this warning,

This warning message is indicating the Embedded Flash/SD-Card (esxi embedded install) is no longer available to the ESXi host.  As this a HP ProLiant server so logged into iLO, checked into diagnostics and found the SD-Card status ok then taken a look into ilo logs and found SD-card was restarted recently.

The good news is that the whole ESXi OS loads into memory so there was no outage for the VMs and once the connectivity would restore the host can access the storage again. The bad news is that the error did not clear automatically and as no one likes to see errors/warning in their production environment so I needed to find a solution to this issue.

The simplest solution of this issue is to put the host in maintenance mode and restart the management agents. One can do this by two ways, either connect to the host using ssh and run below commands,

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

Or alternatively connect to the host using iLO, establish a remote connection, login to DCUI and restart the management agents.

                                       

Once the managements agents restart will complete, vCenter will show the host back in a normal state.

Note: There might be cases where SD-Card having issues due to buggy firmware and in order to fix the issue you may need to upgrade or downgrade the firmware.

we are at firmware version 2.20 and as per various forums this version have SD-Card related bug and that was supposedly fixed in firmware version 2.22, as version 2.30 is also available so one may upgrade to one of these versions of firmware.

Other Scenario: What if SD-Card is failed, you can try to remove and reattach the SD-Card but if it still doesn’t come online then you need to call the server vender for its replacement. 

But if SD-Card is bad, migrate all VMs to other hosts then put the host in maintenance mode and take backup of host configuration. Now shut down the host and after replacing the flash drive reinstall the esxi (As the host will not come up after reboot), once the host comes up, configure the management network and VLANs then restore the host configuration.

Reference: Daniel's blog and discussion on other forums.

Update, 05/11/2015:- This week we faced the same issue again so instead of fixing it myself contacted HP support and they confirmed the issue is with firmware version 2.20 that we have on these G9 server.

Response from hp support:  That version 2.20 has been removed from our site due to it causing issues with server components, including the embedded flash cards. . The new iLO firmware 2.22 addresses/fixes issues with the embedded cards disconnecting.

  

And hp support further suggested us to upgrade the firmware of G9 servers to most recent version 2.30.

That’s it… :)

vCenter Server 5.5 installation/upgrade stuck at Orchestrator installation screen

Last day when I was re-installing vCenter server in our test environment, came across this issue where vCenter install process stuck at Orchestrator installation screen for indefinite time (waited for around two hour). 

Initially I thought there might be something wrong with my machine as I had not rebooted the system after vCenter uninstall so I clicked on cancel to end the installation without success then I had to kill the installation process from task manager.
Once the installation terminated, I uninstalled the SSO, inventory service and Orchestrator (yes it was there in add remove program) and rebooted the system. After reboot installed SSO, inventory service and then initiated the vCenter installation again but to my surprise, vCenter installation stuck on same Installing Orchestrator screen again. Then I started Googling about this issue and found a VMware KB article about the same.
As per KB# 2051216,

This is a known issue affecting vCenter Server 5.1 Update 1 and vCenter Server 5.5

To work around this issue:

1. In the Windows Task Manager, locate the vRealize Orchestrator (formerly known as VMware vCenter Orchestrator) processes. For example, you may see processes similar to:
   
   vCenterOrchestrator.exe *32 InstallAnywhere Self-Extractor
   vCenterOrchestrator.exe *32 LaunchAnywhere
2. End the process tree for the vCenterOrchestrator.exe *32 LaunchAnywhere process.
   
   The vCenter Server installer should now complete successfully.

Note: If you are a user of vRealize Orchestrator (formerly known as VMware vCenter Orchestrator), you can complete the configuration manually through the vRealize Orchestrator configuration service.

Related KB article# 2045613.

That's it... :)

Looking up MoRef (Managed Object Reference) IDs in vSphere environment

Today I saw a question about VM MOID like how to get it. The answer could be pretty simple like using RVTools,

But what are the other ways, before talking about that, first understand what this MoRef ID is. Every object within a vSphere environment is internally tracked, and has a unique identifier called moRef ID. A Managed Object Reference ID also known just as MoRef ID is a unique value that is generated by the vCenter Server and is guaranteed to be unique for a given entity in a single vCenter instance.
This identifier is composed of a prefix stating the object type, followed by a numerical ID, for example:

host-25
vm-11
datastore-09   etc.

You can check the MoRef ID of a host by going to this URL:
https://vC_Server FQDN_or_IP/vod/index.html

Now we will look up the specified moRef ID using Managed Object Browser it will show you the corresponding object with all its properties, including its name of course.for other objects. Go to https://vC_Server FQDN_or_IP/mob/?moid=host-id

From here you will get the MoRef IDs of other objects and can get further detail about the by using managed object browser.

To check moRef IDs using PowerCLI, for this connect to your vCenetr or a host using using the Connect-VIServer cmdlet,
Connect-VIServer <vcenter_or_host> -User uaser_name -Password *****

After you have successfully connected, the Get-<object_type> cmdlets allow you to know which object has a particular MoRef ID in no time,
Get-VMHost | ft -Property Name, ID

The “ID” field returned by PowerCLI is actually the moRef, preceded by a slightly more “verbose” string stating the object type.
Replace Get-VMHost with  Get-VM or Get-Datastore to carry out equivalent operations on other object types

You can always search for a specific ID, just be sure to add an asterisk (*) before the moRef ID you are looking for so you don’t have to include the object type string every time.
Get-VM -ID *moRef_id | ft -Property Name,ID                                        or
Get-Datastore -ID *moRef_id | ft -Property Name,ID -AutoSize

Reference: Danilo Chiavari and William Lam's blog posts.

That's it... :)