How to capture memory dump of a VM from snapshot or suspended state file

This is something could be requested by an application vendor for debugging purpose to investigate an application related issue. If you get any such request for VM memory dump then you might wonder how to capture a memory dump from a VMware virtual machine without stopping its execution.

If this is a production VM then you might not want to force a crash or change Windows dump parameters and reboot the machine. So, how can we capture a memory dump of a VM without interrupting it.

There is a VMware fling called vmss2core, using which we can convert the checkpoint state files into formats that third party debugger tools understand. It can handle both suspend (.vmss) and snapshot (.vmsn) checkpoint state files as well as both monolithic and non-monolithic (separate .vmem file) encapsulation of checkpoint state data.

The vmss2core tool can produce core dump files for the Windows debugger (WinDbg) as well as for other operating systems. Please refer to given screenshot for more info.

For more info about usages of vmss2core tool, please refer to Debugging Virtual Machines with the Checkpoint to Core Tool

We need to take a snapshot of affected VM when it hangs, crashes, or otherwise display symptoms you are troubleshooting and then download the snapshot state file (.vmsn) and VM paging file (.vmem) which can later be converted to Windows memory dump file (.dmp) using vmss2core utility.

Steps:
1. Copy the vmss2core.exe utility to the same location where you downloaded the VM snapshot or suspended state files (.vmsn – in case of snapshot or .vmss for suspended state).

2. Open a command line and navigate to the location of the snapshot / suspended state files and execute the following command:

Here you would use the tool with OS‐specific options. For example, this command generates a memory.dmp file for the Windows debugger, WinDbg.

For Snapshot:
C:\folder>vmss2core.exe -W snapshot.vmsn [snapshot.vmem]

For Suspended state:
C:\folder>vmss2core.exe -W snapshot.vmss 

If the snapshot file is from a Windows 8 or Windows Server 2012 VM, use

C:\folder>vmss2core.exe -W8 snapshot.vmsn [snapshot.vmem]

Successful output of this should be a "memory.dmp" file suitable for use with WinDbg

Please note: VM paging file (.vmem file) may not be present depend on the state of VM.

Related VMware kb# 2003941 

That's it... :)

Fixing the error "vMotion is not enabled on the host of the Virtual Machine"

This is first time when I saw such error on a production host, which I know have everything properly configured (I mean, identical vmkernel port with vMotion enabled) and moreover vMotion was working for for this VMhost before.

When I saw this error at first, thought someone probably disabled vMotion inadvertently however on cross-checking found everything correct.

So, this is something happening due to probably some kind of technical glitch and to fix this we either need to reboot the host or simply disable/re-enable the vMotion in vmkernel port properties.

You can disable/re-enable the vMotion by editing respective vmkernel port but make sure you give it a few minutes before re-enabling.

That’s it…. 😊

"The disk is write protected" error on a Windows Failover Cluster Node VM

Lately, I came across this issue where database team was unable to start the sql service on SQL cluster nodes and when they checked SQL logs, found the drive where temp database stored was no more writable (I have seen this issue a few times in past and if I remember correctly every time this happened on Windows Server 2008 R2 server failover cluster nodes) .

When tried to create a new object in this drive, found no option to do so and checking the disk for errors from disk properties => Tools, end up with an error like,

I am not sure what caused this issue however, found following Volume Shadow Copy service error entries in event viewer, “A critical dynamic disk is a virtual hard disk. This is an unsupported configuration.” So, suspect this has something to do with volume shadow copy and Microsoft server cluster.

We can fix this Write protected disk issue by clearing the read only attribute from volume level (in my case clearing the read only attribute from disk didn’t work so suggest you clear the attribute at lowest level).

To do so follow these steps:

First open disk management and note down the disk and volume name/number of affected drive/volume.

1. Open a command prompt, type in Diskpart and then press Enter.
2. Run the command “List Volume” and press Enter.
3. Now decide the Volume name, dive letter (Affected Read only volume, identified earlier from Disk Management).
4. Select the affected volume by using “Select volume x” command, where x is the volume number
5. One the affected volume is selected, now to clear the “Read-only” attribute, run the command “attributes volume clear readonly”. 

                       

And with this you are done, now your disk/volume should be writable.

You can check the same by running “detail volume” or “Attributes volume” command.

Note: Here one may ask here why I didn’t verify the disk/volume read/write attribute earlier and the short answer is yes, I did check and interestingly readonly attribute was set to No but still I have to clear the readonly attribute to make it writable again 😉.

In Case if your folders inside the drive are still not writable then run following cmd to remove the Read-only attribute and to set the System attribute.

C:\attrib -r +s drive:\folder_name 

Hope this would be helpful to others.........That's it :)

HP SPP upgrade failing with unexpected errors

This week while upgrading server Firmware/Drivers on HP ProLiant G9 servers using latest HP SPP, on one server SPP online deployment return with following,

And when we checked install logs found all updates failed. As lately I had faced an issue where HPSUM inventory was failing due to missing HP management tools and as per my past experience I was aware about the fact, VMware update installations or individual vib install may fail if server has no esxi image profile attached/image profile got corrupted hence thought of cross checking the same.

On checking found this server had no host image profile defined,

When checked the esxupdate.log file found entries of the name of HPE custom esxi 6.0 u2 install file that mean the install was done using the correct ISO image and somehow it got corrupted later.

Then checked for the available image profiles by searching for Imgdb.tgz file (this is the image profile backup and by looking at its size we can get an idea if the image profile got corrupted), this will list the two Imgdb.tgz files as follows,

By looking at the size of these files, size of one imgdb.tgz file is comparatively very less and seems got corrupted.

We can fix this broken image profile issue by either doing an upgrade or fresh install. There is one more way to fix the above issue and that is by copying/replacing the host image profile of this host with an image profile from a working host.

We can do that as follows,

Use winscp or any other ftp client to connect to any healthy Esxi host and browse to /bootbank dir and copy the Imgdb.tgz file to your system and now connect to the host that is missing the image profile and paste the earlier copied Imgdb.tgz file to /tmp dir.

Now

Remove the corrupted imgdb.tar file,

rm /bootbank/Imgdb.tgz

Extract the Imgdb.tgz dir copied from healthy host,

tar -xzf /tmp/Imgdb.tgz

then copy the working image profile and related vibs to /var/db/esximg/profiles and /var/db/esximg/vibs directories,

cp /tmp/var/db/esximg/profiles/* /var/db/esximg/profiles/

cp /tmp/var/db/esximg/vibs/* /var/db/esximg/vibs/

After this run following cmd to create backup of image,

/sbin/auto-backup.sh

Now when you check for the installed profile, it will show you one. It wouldn’t reflect the host profile name in host summary until you re-start the host mgmt agents or reboot the host.

After doing above, when I re-initiated the SPP online update, this time host firmware/driver got updated without any further issue.

Related blog post: http://www.provirtualzone.com/esxi-6-0-fix-corrupted-host-imageprofile/

That’s it… J

HP SPP online update: HPSUM host inventory failed with an error like re-enter the credentials

Lately, I was upgrading HP servers firmware/drivers using HP SPP online upgrade method and for one HP ProLiant server after adding the node to HP SUM inventory when I try to inventory my box, got an inventory error and it asks me to re-enter my credentials while the entered credentials were correct.

Usually when a node is added you must inventory it against the SPP repository so HPSUM could compare and determine what is needed for your box.
When googled about the issue, most of the results pointed out this host seems missing some HPE VIBs that must be on the box prior to attempting to use HP SUM, this was strange as this host was installed using the HPE provided custom ESXi 6.0 U2 ISO image.

Here in order to fix the issue, downloaded and installed the HPE custom ESXi 6.0 U2 image bundle and after that there was no issue with SPP update (as this also installed all missing HP management components).

To upgrade the esxi install using VMware offline update bundle zip use,

 esxcli software vib update -d /vmfs/volumes/…./vmware_update_offline_bundle.zip

There are other ways too to upgrade the esxi host, like installing one of the available profiles in offline update bundle.

To list the installed image profile we can use following cmd,

esxcli software profile get

Now list the available image profile in download VMware update offline bundle,

 esxcli software profile list -d /vmfs/volumes/…./vmware_update_offline_bundle.zip

The output of above cmd would list the available image profiles from the downloaded vmware_update_offline_bundle.zip, note down the name of image profile you want to install, now

 esxcli software profile install -d /vmfs/volumes/…./vmware_update_offline_bundle.zip -p vmware_inage-profile name

This will install/update the desired esxi image profile.

That’s it… J

AWS Certified SysOps Administrator - Associate exam experience and few exam related tips

I sat the AWS Certified SysOps Administrator - Associate exam today and passed, in this post I would share my exam experience and a few exam related tips.

There were 55 questions in this exam and and most of the questions were quite trickier than AWS Solution Architect - Associate exam and I hardly found any definitive question however that doesn't mean you would not read and understand definitions of different services ;) 

There were scenario based questions about changing tenancy of a running instance, placement group, EBS snapshots, Instance/ephemeral storage, ELB, Auto-scaling, Route53 records, Routing policy, IAM policies, S3 storage related backup scenario, network connectivity related questions in VPC, Cloudwatch alarms, health checks, Multi-AZ RDS & Read replica, Elastic beanstalk, Cloud Formation, Opswork and AWS shared responsibility model etc.

Here is my exam score, Overall Score: 83%

Topic Level Scoring:
1.0 Monitoring and Metrics: 87%
2.0 High Availability: 100%
3.0 Analysis: 55%
4.0 Deployment and Provisioning: 87%
5.0 Data Management: 85%
6.0 Security: 75%
7.0 Networking: 100%

About the preparation of this exam, learn ins and out of all the AWS services covered in this exam by heart and the best option to do that is, watch this certification's training CBTs few times and once you are done with that now review the FAQ of these services and read AWS security withe paper once. 

I used Ryan Kroonenburg aka AcloudGuru's ( AWS Certified SysOps Administrator - Associate  online ILVT course and Elias Khnaser's AWS Certified SysOps Administrator - Associate course on Pluralsight however i must say any one of these course is suffice to pass the exam.

I would also suggest going for AWS solution architect course/exam first, as it would give you a better understanding about most of the AWS services and provide a solid base for this SysOps exam.

My Previous post about AWS Certified Solution Architect exam experience and few exam related tips can be found here.

Finally before going for the exam, make sure you also practiced on various sample questions available over web, however don't depend on then them as you would find many wrong answers there (however you will be able to to catch them only if you know the ins and out of different serveries by heart).

Hope this would be helpful in passing you AWS Sysops exam....Best of Luck!

That's it.... :)

Windows Server reboot/shutdown events in event viewer

Ever found yourself wondering about an unexpected system reboot, these event IDs are very useful while one is investigating the cause of unexpected system shutdown/reboot. 
I not sure about others but but always found hard remembering these event IDs so making a note for future reference and believe others will also find it useful.

Server reboot/shutdown events:

•  Event ID 6005: “The event log service was started.” This is synonymous to system startup.
•   Event ID 6006: “The event log service was stopped.” This is synonymous to system shutdown.
•   Event ID 6008: "The previous system shutdown was unexpected." Records that the system started  after it was not shut down properly.

•   Event ID 6009: Indicates the Windows product name, version, build number, service pack number,   and operating system type detected at boot time.
•    Event ID 6013: Displays the uptime of the computer. There is no TechNet page for this id.
•    Event ID 1074: "The process X has initiated the restart / shutdown of computer on behalf of user     Y for the following reason: Z." Indicates that an application or a user initiated a restart or                  shutdown.
•    Event ID 1076: "The reason supplied by user X for the last unexpected shutdown of this computer    is: Y." Records when the first user with shutdown privileges logs on to the computer after an           unexpected restart or shutdown and supplies a reason for the occurrence.

Note: In case of unexpected shoutdown due to power failure, there would be no event created.

To know the system boot Time:

  C:\systeminfo | find /i “boot time”

That's it... :)

Microsoft eBook Giveaway: Millions of Microsoft e-books available for free download

This is the time of year again when Microsoft is giving away millions of e-books for free, this includes books for Windows servers, SQL Server, SCCM, exchange, PowerShell, Azure Cloud and etc.

Please make use of this opportunity and download any required e-book.

Checkout below MSDN blog for the list of available books/download links,

https://blogs.msdn.microsoft.com/mssmallbiz/2017/07/11/largest-free-microsoft-ebook-giveaway-im-giving-away-millions-of-free-microsoft-ebooks-again-including-windows-10-office-365-office-2016-power-bi-azure-windows-8-1-office-2013-sharepo/

Happy learning… J

PowerCLI script to create multiple PortGroups on a newly added VMhost

This script is useful when you are introducing a a new host to your environment and need to create multiple VLANs on this host.
Both of these script would do the same thing but the only difference is, to make use of first script you first need to create a csv file having VirtualSwitchName, VLANname, VLANid details on the other hand second script would take one host of your choice as reference and would create the same VLANs on target host.
Script 1: -

<# ==================================================================

Title: create_multiple_vlan-2.ps1

Description:this script can be used to create VLANs on a newly added VMhost, it will read VLAN-detail.csv and create VM port groups

Requirements: Windows Powershell with PowerCLI installed

Pre-req: You need to create a vlan-detail.csv with required VirtualSwitchName, VLANname, VLANid, and the the vSwitch should be present on target host

Usage: .\create_multiple_vlan.ps1

==================================================================== #>

Add-PSSnapin VMware.VimAutomation.Core

$myvc = read-host -Prompt "Enter your vCenter server Name or IP name"

Connect-VIServer $myvc

$targetVMhost = read-host -Prompt "Enter your target host name"

$InputFile = “c:\vlan-detail.csv”

$MyVLANFile = Import-CSV $InputFile

ForEach ($VLAN in $MyVLANFile) {

$MyvSwitch = $VLAN.VirtualSwitchName

$MyVLANname = $VLAN.VLANname

$MyVLANid = $VLAN.VLANid

get-vmhost $targetVMhost | Get-VirtualSwitch -Name $MyvSwitch | New-VirtualPortGroup -Name $MyVLANname -VLanId $MyVLANid

}

disconnect-VIServer $myvc -Confirm:$false

Script 2 :-

<# ==================================================================

Title: create_multiple_vlan.ps1

Description:this script can be used to create VLANs on a newly added VMhost by taking referance of any existing host

Requirements: Windows Powershell with PowerCLI installed

pre-req: vSwitch should be present on target gost

Usage: .\create_multiple_vlan.ps1

==================================================================== #>

Add-PSSnapin VMware.VimAutomation.Core

$myvc = read-host -Prompt "Enter your vCenter server Name or IP name"

Connect-VIServer $myvc

$sourceVMhost = read-host -Prompt "Enter your source host name"

$targetVMhost = read-host -Prompt "Enter your target host name"

get-vmhost $sourceVMhost | Get-VirtualSwitch | Get-VirtualPortGroup | select VirtualSwitchName, Name, vlanID | export-csv "c:\vlan-detail.csv" -NoTypeInformation

$InputFile = “c:\vlan-detail.csv”

$MyVLANFile = Import-CSV $InputFile

ForEach ($VLAN in $MyVLANFile) {

$MyvSwitch = $VLAN.VirtualSwitchName

$MyVLANname = $VLAN.name

$MyVLANid = $VLAN.VLANid

get-vmhost $targetVMhost | Get-VirtualSwitch -Name $MyvSwitch | New-VirtualPortGroup -Name $MyVLANname -VLanId $MyVLANid

}

Remove-Item $InputFile

disconnect-VIServer $myvc -Confirm:$false

Hope these scripts would be useful.

That's it... :)

AWS Certified Solution Architect exam experience and few exam related tips

Last week I attempted and passed my AWS Certified Solution Architect exam, in this post I would share my exam experience and a few exam related tips.

I have never been as anxious and scared as I was before this exam and the obvious reason was not having much hands on experience on AWS platform and the exam feedback related posts which I read on different AWS forums where people are talking about how though these AWS exams are however once the exam started and after looking at first few questions all nervousness instantly went away. Now After passing this exam I can say the questions were very fair neither lengthy nor very hard and what I mean by this is, if you study all the AWS services which are covered in this exam and understand the in and out of them, then you are going to pass this exam easily.

Here is my exam score, Overall Score: 98%

Topic Level Scoring:
1.0 Designing highly available, cost efficient, fault tolerant, scalable systems : 96%
2.0 Implementation/Deployment: 100%
3.0 Security: 100%
4.0 Troubleshooting: 100%

I was preparing for AWS CSA as well as Sysops certification from last three - four months and during my preparation I watched ACloud.guru and Plurasight.com CBT videos two – three times as well as read about different AWS service guides on AWS docs site few times times.   

Services I focused for this particular exam : 

S3	IAM	EC2	Storage Gateways	VPC
Route53	CloudWatch	CloudTrail	CloudFront	ELB
CloudFormation	AutoScalling	ElasticBeanstalk	RDS	DynamoDB
SQS	SNS	SES	Redshift

This exam had 60 questions and out of which around 30 questions were scenario based, where you would be tested about your knowledge of different AWS services…Which service fits best where.  

Beside above listed services, read and understand following one page docs, you would find scenario based questions related to these topics  

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html

http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html

https://aws.amazon.com/premiumsupport/knowledge-center/snapshot-ebs-raid-array/

http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Security.html

https://d0.awsstatic.com/whitepapers/aws-security-best-practices.pdf ,   Page 2

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-comparison.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html

http://cloudacademy.com/blog/aws-bastion-host-nat-instances-vpc-peering-security/

https://aws.amazon.com/premiumsupport/knowledge-center/instance-store-vs-ebs/

http://docs.aws.amazon.com/emr/latest/DeveloperGuide/emr-kinesis.html

https://aws.amazon.com/directconnect/   

And most importantly don’t forget to read the FAQ of S3, VPC, EC2, Route53 and RDS/DynmoDB as well as Security related whitepaper.

I also found this post on A cloud guru very useful, https://acloud.guru/forums/aws-certified-solutions-architect-associate/discussion/-KSDNs4nfg5ikp6yBN9l/exam_feedback_-_20_specific_po

Hope this would be helpful in passing your AWS CSA exam....Best of Luck!

That's it.... :)

vCenter VM operations error: The operation is not allowed in current state

Today while trying to take snapshot of one the VM, end up with following error,

“The operation is not allowed in current state”

Then I tried to vMotion the VM to any other host but no luck, same was true for any other task however I didn’t see any issue with other VMs on this host.

I also checked for any associated running task but couldn't find any.

I had seen such errors earlier and most of the time such error occur because either the ESXi host or a virtual machine is in an unexpected state and to work around this, we just need to restart the host management agents.

Note: Before restarting the host management agents, make sure no Virtual machine is configured to poweron/off with host otherwise it may reboot.

To restart the VMware host management agent, connect to host over ssh using putty and run following commands,

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

Reference: Related VMware kb# 1003829

That’s it… J

VMware vExpert 2017 Announced

VMware just announced the vExpert 2017…. I am very honored to be named a VMware vExpert again…..Congrats to all those who named as a 

Here is the full list of vExpert 2017... https://blogs.vmware.com/vmtn/2017/02/vexpert-2017-award-announcement.html 

That's it... :)

vimtop: To Monitor the performance and Resource Usage of services on vCenter Server Appliance

Vimtop is a tool similar to esxtop, which runs in the environment of the vCenter Server Appliance. By using the text-based interface of vimtop in the appliance shell, we can view the performance information and statistics about VCSA and the applications running under it and a list of vSphere services and their resource usage.

If you have ever used esxtop before with ESXi, then you would have a similar experience here. This will definitely be a handy a tool to be aware of when needing to troubleshoot performance issues or bottlenecks in the VCSA as useong vimtop we can monitor vCenter services in real time.

To launch vimtop, we first need access Appliance direct console or SSH to a VCSA 6.x system and type "vimtop" in either the appliance shell or in a regular bash shell.

To access the appliance shell, press ALT+F1 when you are accessing Appliance console (same like accessing DUCI of ESXi).

As you can see, when it launches, shows combination of cpu and memory related information and also the processes running on the appliance with resources used. Notice at the top we see the uptime, load averages, cpu breakdown, memory and swap related information.

As highlighted in last image, on this screen we can access Help by pressing H, Disk related info using K, Network info using O, Pause and refresh by pressing P and R keys etc.

When running in interactive mode, vimtop recognizes several other single-key command

All interactive mode panels recognize the commands listed on this page.

For detailed info about vimtop usage, please have a look at vimtop related topics in VMware Documentation Here.

That's it... :)

PowerCLI script to get list of Snapshots older then x days as a mail atachment

This script would sent you a list of Snapshots older than x days(replace x with the the number of days as required) in html table format as a mail attachment.

To run this script, copy the content of it to notepad file and save it as file_name.ps1 now you can run it from any system which have Windows Powershell and VMware PowerCLI installed.

<# ==================================================================
Title: Get_Snapshot_list.ps1
Description: List Snapshots older than x days and then email it as an attachment
Requirements: Windows Powershell with PowerCLI installed
Usage: .\Get_Snapshot_list.ps1
==================================================================== #>
Add-PSSnapin VMware.VimAutomation.Core

$viserver = Connect-VIServer vCenter_Server_Name_or_IP -user "userName" -password "your vCenter Password"

$SnapshotReport = Get-VM  | Get-Snapshot | where { $_.Created -lt (Get-Date).AddDays(-15)} | select VM, Name, Powerstate, SizeMB, Created | ConvertTo-Html | Set-Content SnapshotReport.html

$EmailFrom = "from_mail_id"

$EmailTo = "to_mail_id"

$subject = "Snapshots older then x days"

$body = "Please find the enclosed snpshot report: SnapshotReport.html"

$smtp = "smtp_server_address"

$viserver; $SnapshotReport; Send-MailMessage -From $EmailFrom -To $EmailTo -Subject $subject -Body $body -SmtpServer $SMTP -Attachments "snapshotreport.html" -Priority "High"

Remove-Item SnapshotReport.html

Disconnect-VIServer "vCenter_server_name_or_IP" -Confirm:$false

By scheduling this script using task scheduler we can get the snapshot report on weekly basis. To schedule this script, please check the following post, How to run Powershell script as a scheduled task.

As here the login credentials are written in plain text so if are planing to schedule this script then instead of using login credentials in script, it would be better to schedule it using credentials which already have login rights on vCenter and as PowerShell supports passthrough authentication so that will also work.

That's it... :)

VMware vCenter Server 6.x Appliance services: how to find svc status or start/stop

In VMware vCenter Server 6.0 and later, VMware recommends to use the vSphere Web Client or Service Control command-line tool to check, stop, start, or restart vCenter Server Appliance services.
Status of vCenter appliance services can be checked either from vCSA appliance shell or by using web client.

Note: To access vCSA appliance shell, same like ESXi, access to the appliance console and press Alt+F1, we can also access appliances shell over ssh using putty but for that we should have enabled ssh first and that can be done either from vCSA direct console or from web client console.

Listing the vCenter Server Appliance services

To list the vCenter Server Appliance services within the vSphere Web Client:

1. Log in to the vSphere Web Client with a vCenter Single Sign-on administrator account.
2. Navigate to Administration > Deployment >  System Configuration.
3. Click Nodes, select the vCenter Server Appliance node and click the Related Objects tab.

To list the vCenter Server Appliance services using the command-line:

1. Log in as root through an SSH or console session on the vCenter Server Appliance.
2. Run this command to enable the shell:
   
   shell.set --enabled true
3. Run this command to launch the shell:
   
   shell
4. Run this to change directories to /bin:
   
   cd /bin
5. Run this command to list the vCenter Server Appliance services:
   
   service-control --list

6.  To view the current status of the vCenter Server Appliance services, type the command:

service-control --status

To check the status of any specific service, for ex. vmware-vpxd

service-control --status vmware-vpxd

To Start / Stop or Restart a vCenter Server Appliance services

To start/stop/restart the vCenter Server Appliance service using the vSphere Web Client:

1. Log in to the vSphere Web Client with a vCenter Single Sign-on administrator account.
2. Navigate to Administration > Deployment >  System Configuration.
3. Click Nodes, select the vCenter Server Appliance node and click the Related Objects tab.
4. Right-click on the service you would like to start/stop/restart and select desired option.

To start the vCenter Server Appliance service if it is was stopped using the command-line:

1. Log in as root through an SSH or console session on the vCenter Server Appliance.
2. Run this command to enable the shell:
   
   shell.set --enabled true
3. Run this command to launch the shell:
   
   shell
4. Run this command to change directories to /bin:
   
   cd /bin
5. Run this command to list the vCenter Server Appliance services:
   
   service-control --list

Now, Run this command to start a specific service:

service-control --start servicename

or

Run this command to stop a specific service:

service-control --stop servicename

You may also start all services by typing the command:

service-control --start --all

or

Stop all services by typing the command:

service-control --start --all

To perform a dry run of the command, add the option --dry-run to the command, doing so will display what actions the command will run without executing the actions. For example, type the command:

service-control --stop --all --dry-run     or   service-control --start --all --dry-run

Note: To restart a vCSA appliance server you first need to stop and then start the services as there is no restart switch via shell cmd.

Reference: VMware kb# 2109887, VMware vSphere 6.5 Documention 

Note: For VCSA 5.x appliance services, please refer to kb# 2054085

That's it... :)

How to Reset vCenter SSO admininstrator password in vSphere 6.x

You might have faced this, specially in you homelab env ;) , where after a long time when you tried to login on to vCenter SSO and couldn't recall the SSO administrator account "administrator@vsphere.local" password.

Here the good things is, the process of resting SSO administrator account password is pretty state forward, and we can easily reset the "administrator@vsphere.local" password.

To reset the administrator@vsphere.local password:

On a Windows Platform Services Controller or vCenter Server with Embedded Platform Services Controller:

1. Log in to vCenter Server with a domain administrator account. If the Platform Services Controller is installed separate from vCenter Server, log in to the Platform Services Controller server.
2. Open an elevated command prompt.
3. Run C:\> "%VMWARE_CIS_HOME%\vmdird\vdcadmintool.exe".
   
   This console loads:
   
   ===============================
   Please select:
   0. exit
   1. Test LDAP connectivity
   2. Force start replication cycle
   3. Reset account password
   4. Set log level and mask
   5. Set vmdir state
   ===============================


4. Press 3 to enter the Reset account password option.
5. When prompted for the Account UPN, enter: administrator@vSphere_Domain_Name.localBy default, this is: administrator@vSphere.localA new password is generated.

Notes:

• If you customized your vSphere Domain name, provide the customized domain name.
• If the prededing steps fail with a domain administrator account, use a local administrator account.

    6. Use the generated password to log in to the administrator@vSphere.local account.
    7. After the password is regenerated, log in to vSphere Web Client and change the password.

On the Platform Services Controller or vCenter Server with Embedded Platform Services Controller Appliance

1. Log in to vCenter Server Appliance using SSH as the root user.
2. Run this command to enable access the Bash shell:
   
   shell.set --enabled true
3. Type shell and press Enter.

      4.  Now run /usr/lib/vmware-vmdir/bin/vdcadmintool

        

        5. Press 3 to enter the Reset account password option.

        6. When prompted for the Account UPN, enter: administrator@vSphere_Domain_Name.local

            By default, this is: administrator@vSphere.local

Once you press enter, a new password is generated.

Note: If your vSphere Domain name is customized, provide the customized domain name.

      7. Use the generated password to log in to the administrator@vSphere.local account.
      8. Once you log in to the vSphere Web Client, change the password.

Reference: VMware kb# 2034608

That's it... :)