To apply the workaround for CVE-2021-44228 to vCenter Server 7.x, vCenter 6.7.x & vCenter 6.5.x there are workaround sections to update on the vCenter Appliance.
Note: For vCenter Cloud Gateway, only the steps for the vMon Service and Analytics Service are necessary.
Click here for vCenter Server Appliance 6.7.x workaround
Click here for vCenter Server Appliance 6.5.x workaround
Click here for vCenter Server Appliance 6.0.x workaround
vCenter Server Appliance 7.0.x Workaround
vMON Service
- Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
- Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
- At the very bottom of the file, replace the very last line with 2 new lines
Execute this step number ‘3’ based on the vCenter version running in your environment.
NOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-
- vCenter 7.0 Update 3, 3a
- vCenter 7.0 Update 2, 2a, 2b, 2c, 2d
Original
exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_argsNOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-
- vCenter 7.0 GA, 7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d
- vCenter 7.0 Update 1, U1a, U1c, U1d
Original
exec $java_start_bin $jvm_dynargs “$@”Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
- Ensure the file permissions are set correctly with the below commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
- Restart vCenter Services
service-control –stop –all
service-control –start –all
Update Manager Service
- Back up the existing start.ini file
cp -rfp /usr/lib/vmware-updatemgr/bin/jetty/start.ini /usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
- Update the start.ini file
vi /usr/lib/vmware-updatemgr/bin/jetty/start.ini
- Add the following line to the end of the file
-Dlog4j2.formatMsgNoLookups=true
- Restart the Update Manager Service
service-control –restart vmware-updatemgr
Analytics Service
- Back up the log4j-core-2.8.2.jar file
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Restart the Analytics service
service-control –restart vmware-analytics
DBCC Utility
- Back up the log4j-core-2.8.2.jar file
cp /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Note: This command may fail with “zip error: Nothing to do!”. If so, proceed to the verification section to ensure the step was not needed. No service needs to be restarted for DBCC.
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the vMon services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the Update Manager changes are shown under “System Properties” in the output of the following two commands:
cd /usr/lib/vmware-updatemgr/bin/jetty/
java -jar start.jar –list-config
System Properties:
——————
log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)
- Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
This should return 0 lines
- Verify the DBCC Utility changes:
grep -i jndilookup /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar | wc -l
This should return 0 lines
vCenter Server Appliance 6.7.x Workaround
vMON Service
- Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
- Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
- At the very bottom of the file, replace the very last line with 2 new lines
- Restart vCenter Services
service-control –stop –all
service-control –start –all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
Analytics Service
NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only. The JAR is already updated to 2.11 on the later versions.
- Back up the log4j-core-2.8.2.jar file
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Restart the Analytics service
service-control –restart vmware-analytics
CM Service
- Back up the log4j-core.jar file
cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Restart the CM service
service-control –restart vmware-cm
Secure Token Service
- Back up and edit the the vmware-stsd file
cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
- Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
start_service()
{
perform_pre_startup_actionslocal retval
JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
- Restart the vmware-stsd service
service-control –stop vmware-stsd
service-control –start vmware-stsd
Identity Management Service
- Back up and edit the the vmware-sts-idmd file
cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
- Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
- Restart the vmware-sts-idmd service
service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
This should return 0 lines
- Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
This should return 0 lines
vCenter Server Appliance 6.5.x Workaround
vMON Service
- Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
- Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
- At the very bottom of the file, replace the very last line with 2 new lines
- Restart vCenter Services
service-control –stop –all
service-control –start –all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
CM Service
- Back up the log4j-core.jar file
cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Restart the CM service
service-control –restart vmware-cm
Secure Token Service
- Back up and edit the the vmware-stsd file
cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
- Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
start_service()
{
perform_pre_startup_actionslocal retval
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
- Restart the vmware-stsd service
service-control –stop vmware-stsd
service-control –start vmware-stsd
Identity Management Service
- Back up and edit the the vmware-sts-idmd file
cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
- Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
- Restart the vmware-sts-idmd service
service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd
PSC Client Service
- Back up and edit the vmware-psc-client file
cp -rfp /etc/rc.d/init.d/vmware-psc-client /root/vmware-psc-client.bak
vi /etc/rc.d/init.d/vmware-psc-client
- Insert a new line near line 300, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Djava.io.tmpdir=”$CATALINA_BASE/temp” \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
- Restart the vmware-psc-client service
service-control –stop vmware-psc-client
service-control –start vmware-psc-client
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
This should return 0 lines
vCenter Server Appliance 6.0 U3j Workaround
vCenter Server Appliance 6.0 U3j is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 due to the Performance Charts service. Mitigation steps have been identified as follows:
- Back up and edit /usr/lib/vmware-perfcharts/wrapper/conf/wrapper.conf on the appliance and add a new line just below “wrapper.java.additional.13=-Dlog4j.configurationFile=file:/etc/vmware-perfcharts/log4j2.xml” (line 72) with the following content:
wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true
- Save the file, stop the service and then start it through service-control:
service-control –stop vmware-perfcharts
service-control –start vmware-perfcharts
Note: vCenter Server Appliance versions 6.0GA – 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-core-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-api-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-core-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-api-2.1.jar