High CPU usage in vCenter Server Appliance 6.7

 

 

 

  Symptoms

• Increase of CPU usage after upgrading to vCenter Server Appliance 6.7.
• Slow performance when do operations in vSphere Client (HTML5) and vSphere Web Client (Flex).
• In the Monitor > Performance tab, the CPU usage of VCSA 6.7 may be more than 90%.

Cause

In this case the root cause of high CPU usage in VCSA 6.7 is the Jetty, a Java HTTP server, which is used by the vSphere Update Manager integrated in VCSA. The Open source Jetty has reported multiple issues with high CPU usages. Read more information about this known issue of Jetty from 100% CPU usage in Selector using Jetty

Resolution

This is a known issue and plan to fix in vSphere 6.7 P03.
To workaround the issue, 
1. SSH login to vCenter Server Appliance.
2. Edit the file config.conf:
# vi /etc/vmware-vsan-health/config.conf
3. At the end of this file, add the below content:
[KillSwitch]
VumIntegration = False
4. Restart the service of vmare-vsan-health
# service-control --stop vmware-vsan-health
# service-control --start vmware-vsan-health

How to identity if high CPU usage issue matched the case, please follow the steps:
1. SSH login to VCSA.
2. List the PID of process which consumes the most of CPU
# ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head
  PID  PPID  CMD                          %MEM  %CPU
46852  6953  jre/bin/java -Dhttps.protoc  25.7   771
3. Display the running command information of the process:
# ps axww opid,etime,cmd | grep <PID>
In the above sample, the PID is 46852. The sample cmdlet is:
# ps axww opid,etime,cmd | grep 46852
The output of the cmdlet is similar with below:
 46852  8-03:38:11 jre/bin/java -Dhttps.protocols=TLSv1 -Djavax.net.ssl.trustStore=ssl/vmware-vum.keystore -Djetty.home=jetty/ -Djetty.base=jetty/ -Djetty.port=9084 -Djetty.ssl.port=9087 -Dvum.sdk.port=8084 -Dvum.installdir=/usr/lib/vmware-updatemgr/bin/ -Dvum.patchstore=/storage/updatemgr/patch-store/ -Dvum.patchstore.temp=/storage/updatemgr/patch-store-temp/ -Djava.io.tmpdir=/storage/updatemgr/jetty-temp -Djetty.logs=/var/log/vmware/vmware-updatemgr/vum-server/ -Dlog4j.configuration=jetty/resources/log4j.properties -jar jetty/wrapper.jar jetty-vum.xml jetty-vum-ssl.xml jetty/etc/jetty-https.xml --pre=jetty/etc/jetty-logging.xml

To immediately remediate high CPU usage, you can run below actions:
1. Kill the Jetty process:
# kill -9 PID
2. Restart vmware-updatemgr service
# service-control --stop vmware-updatemgr
# service-control --start vmware-updatemgr