March 2021 Applying MS Exchange 0-Day Patches

On March 2nd Microsoft released Exchange Server Security Updates to address several 0-day exploits targeting Exchange servers. The vulnerabilities, update & mitigations have been covered thoroughly by Microsoft et al. So I’m not going to rehash it here.  If you need additional information, please check the references section at the end of this post. I’ll do my best to keep it updated as the situation is still evolving.

I began planning the update deployment almost immediately, however, there were some rumblings in the community about installation issues.  Several people in this Reddit post described issues, including outright failures and services being left disabled and/or not starting after the installation.

As you might expect, I tested the patch in a lab environment and indeed quite a few services were left in a disabled state.  So, this post is about how I corrected the issue.

I ran the package from the command line with msiexec.exe /Update <Path> /passive /promptrestart.  Despite the arguments the server rebooted without prompting.  After the reboot, quite a few services were stopped and disabled.

DisplayName                                          StartType  Status
-----------                                          ---------  ------
Application Identity                                  Disabled Stopped
Computer Browser                                      Disabled Stopped
IIS Admin Service                                     Disabled Stopped
Internet Connection Sharing (ICS)                     Disabled Stopped
Microsoft Exchange Active Directory Topology          Disabled Stopped
Microsoft Exchange Anti-spam Update                   Disabled Stopped
Microsoft Exchange DAG Management                     Disabled Stopped
...
Microsoft Exchange Unified Messaging                  Disabled Stopped
Microsoft Filtering Management Service                Disabled Stopped
NetBackup SAN Client Fibre Transport Service          Disabled Stopped
Performance Logs & Alerts                             Disabled Stopped
Remote Registry                                       Disabled Stopped
Routing and Remote Access                             Disabled Stopped
ScanMail EUQ Monitor                                  Disabled Stopped
Smart Card                                            Disabled Stopped
SSDP Discovery                                        Disabled Stopped
Tracing Service for Search in Exchange                Disabled Stopped
UPnP Device Host                                      Disabled Stopped
Windows Management Instrumentation                    Disabled Stopped
World Wide Web Publishing Service                     Disabled Stopped

Note: For brevity, some Exchange services were truncated from the above table.

Notice it wasn’t just Exchange services.  For example, IIS AdminService and WMI were both disabled.    From above, I couldn’t tell with certainty which services were disabled by the update installer or what their original start modes were.

To correct this I decided to compare the disabled services to the services on an unaffected Exchange server. On the affected server I ran:

Get-Service | 
Where-Object{ $_.StartType -eq 'Disabled' } |
Export-Csv -Path 'C:\Temp\BadServiceState.csv'

I took that file to an unaffected server and ran:

Import-Csv -Path 'C:\Temp\BadServiceState.csv' |
Get-Service |
Export-Csv -Path 'C:\Temp\GoodServiceState.csv'

Finally, to fix the services, I returned to the troubled server and ran the below loop:

Import-Csv -Path  'C:\Temp\BadServiceState.csv' |
ForEach-Object{ Set-Service $_.Name -StartupType $_.StartType }

At this point, all the startup modes were correct, but I didn’t have a quick way to start the services.  I didn’t want to spend the time tracing out the dependencies to ensure everything would start.  So, I simply let an additional reboot take care of it for me.

After the reboot I reapplied the patch for good measure. This time I ran it via the GUI and had no issues.

To further diagnose the issue I took a quick look at the file C:\ExchangeSetupLogs\ServiceControl.log. The log lists all the services that are stopped and disabled in a format similar to below.

	[08:58:17] Stopping service 'hostcontrollerservice'.
	[08:58:50] Stopping service 'FMS'.
	…
	[08:58:52] Disabling service 'FMS'.
	[08:58:52] Disabling service 'hostcontrollerservice'.
	…

However, the log does a poor job of showing the service configuration prior to the installation.  The process interrogates all services not just those that were changed, making it difficult to parse the file for relevant data. So, instead, I grabbed 2 files from the C:\ExchangeSetupLogs folder while the installer was running.

Apparently these files are used in the last stages of the installation to return service configurations to normal.  Unfortunately, the files are removed at the end of even a faulty install, but if you can grab them during the install you can use a little PowerShell magic to right the ship afterward.

Both files are formatted as Common Language Infrastructure (CLI) XML representations of native PowerShell objects.  In fact, it’s likely these files were created using the Export-CliXml cmdlet.  This is the same type of XML serialization used by PowerShell remoting to communicate objects over the wire.  As such, they are very easy to import and work with in another PowerShell console.

ServiceStartupMode.xml stores an array of hash tables with Name & StartupType keys.  I presume these are used by the installation as splat parameters for the Set-Service cmdlet so one way to leverage the file is:

Import-Clixml <PathTo_ServiceStartMode.xml> | 
ForEach-Object{ Set-Service @_ -ErrorAction SilentlyContinue }

Now, If you reboot the server the services should start the same as before.

Because you’re passing an array of hash tables down the pipeline you can use @_ as the current pipeline element.  Set-Service will treat that as typical splatting.

Note: The file contains information from all services not just the ones modified by the installer.  Since there are some services that can’t be changed, the -ErrorAction SilentlyContinue argument will spare you from profuse error output.

ServiceState.xml stores the actual, albeit serialized ServiceController objects.  As I mentioned before due to service dependencies it would take some work to use this file for corrective action.  However, it may be useful for reporting or other diagnostics.

Of course, attempting to capture these files mid-install is a little inconvenient.  As an alternative PowerShell makes it very easy to capture the same data. You can run the below code to generate the files before running the install package.

Get-Service |
ForEach-Object{
    @{
    Name = $_.Name
    StartupType = $_.StartType
    }
} | 
Export-Clixml -Path c:\temp\ServiceStartupModes.xml

Get-Service | 
Export-Clixml -Path C:\temp\ServiceState.xml

Note: The MS version of the ServiceStartupModes.xml file uses “StartMode” as the key. “StartMode” is an alias for the –-StartupType parameter in the Set-Service cmdlet. An earlier version of this post used “StartType” which is also an alias but only in PowerShell Core 7.x. Ergo, I decided to forego the aliases and use the actual parameter name “StartupType”. However, notice the value is still $_.StartType, the property from any given service.

I also spoke with Microsoft Support and asked them if in future patch releases  they can retain the ServiceStartupMode.xml & ServiceState.xml files in the C:\ExchangeSetupLogs folder.  It’s a simple change that could make a huge difference while working in semi-crisis 0-Day patching scenarios.

MS Support also mentioned some service start issues being linked to missing .DLL files in the /bin folder. They suggested exporting a directory listing of the /bin to a text file. With the export you can use any of a number of methods to isolate missing files and recopy them from a known good server. You can then use the file to figure out which files are missing and copy them back from the known good server.

Quick PowerShell command to export a file listing:

Get-ChildItem "$($env:exchangeinstallpath)bin" -Recurse -Include "*.dll", "*.exe" | 
Select-Object -ExpandProperty FullName | 
Set-Content c:\temp\DLLlist.txt

Hopefully these quick & dirty tricks will help you get through this update cycle a little easier. Feedback is always welcomed, comment, click follow or grab the RSS feed to get notifications of future posts.

Additional Resources Regarding Recent Exchange 0-Day Exploits:

• MS Security Blog: HAFNIUM targeting Exchange Servers with 0-day exploits
• Volexity Blog: Active Exploitation of Multiple Zero-Day Microsoft Exchange Vulnerabilities
• Microsoft Security Response Center (MSRC): Updates Released for Exchange Server
• MS Exchange Team Blog: March 2021 Exchange Server Security Updates
• MS On the Issues Blog: New nation-state cyberattacks
• MSRC: Microsoft Exchange Server Vulnerabilities Mitigations – March 2021
• Microsoft Security Blog: Defending Exchange servers under attack
• MS SUpport: Description of March 2, 2021 Exchange Server Security Update
• MSRC: Update Guide
• GitHub: Hanfium Check Scripts
• Vulnerabilities:
  • CVE-2021-26855
  • CVE-2021-26857
  • CVE-2021-26858
  • CVE-2021-27065
  • CVE-2021-26412 (Not related to known attacks)
  • CVE-2021-26854 (Not related to known attacks)
  • CVE-2021-27078 (Not related to known attacks)