Emergency Fix for Windows Server Crashes Out-of-Band Updates Released by Microsoft

Microsoft has recently released an Emergency Fix for Windows Server Crashes out-of-band update to rectify an identified issue that led to the crash of Windows domain controllers after the installation of the March 2024 Windows Server security updates. This issue could potentially result in severe disruptions to the domain controller’s functioning and hence requires immediate attention.

The update aims to resolve the problem and ensure the smooth functioning of the domain controllers. To avoid any potential complications, it is strongly recommended that businesses and organizations using Windows domain controllers install the update without delay.

As per the community updates, Several system administrators have issued warnings since the latest Patch Tuesday, reporting that servers are crashing and restarting unexpectedly. This is due to a memory leak in the Local Security Authority Subsystem Service (LSASS) process.

Microsoft has confirmed that the latest Windows Server 2012 R2, 2016, 2019, and 2022 updates have known issues affecting all domain controller servers. Microsoft is actively working on a fix for the identified issue.

What’s LSASS Memory Leak

The Local Security Authority Subsystem Service (LSASS) is a critical process that is typically found in Microsoft Windows operating systems. Its primary purpose is to enforce the security policy on the system by ensuring that users who attempt to log into a Windows computer or server are verified. Specifically, the LSASS process handles password changes and generates access tokens, which are used by Windows to grant users access to specific resources on the system.

The Local Security Authority Subsystem Service (LSASS) process has been reported to experience a memory leak issue that results in excessive memory usage and eventual system failure. Several administrators who have encountered this technical anomaly have observed it. The failure of the LSASS process can affect several other system components, thereby significantly impacting overall system performance.

• SUG Software Update Group Patch Deployments SCCM Report using SQL Query
• How to Create Deploy New Software Update Patch Package Using SCCM | ConfigMgr
• How to Add New Patches to Existing Software Update Group|SCCM

Several system administrators reported that, Upon installation of KB5035855 (Server 2016), KB5035849 (Server 2019) and KB5035857 (Server 2022), the lsass.exe process exhibits a significant increase in memory usage. This results in the exhaustion of both physical and virtual memory, rendering the machine inoperable. As such, it is imperative to address this issue promptly to ensure optimized system performance and avoid potential downtime

Temporary Workaround – Install Out-of-Band Updates

Microsoft has released Out-of-Band cumulative updates for Windows Server to fix the LSASS memory leak, preventing servers from crashing and restarting.

Note! Microsoft has not found a permanent solution to resolve this issue. However, Microsoft has realized the need for Out-of-Band Updates. Until Microsoft releases a fix, admins must download OOB updates from the Microsoft Update Catalog and install them on domain controllers affected by the severe memory leak issue.

Operating Systems	Out-of-Band Updates
Windows Server 2022	KB5037422
Windows Server 2019	KB5037425
Windows Server 2016	KB5037423
Windows Server 2012 R2	KB5037426

KB5037422, KB5037425, KB5037423, and KB5037426 are out-of-band releases for Windows Server 2022, Windows Server 2019, Windows Server 2016, and Windows Server 2012 R2 that address a known issue affecting the Local Security Authority Subsystem Service (LSASS). Before installing this update, you must install the latest servicing stack update (SSU). Microsoft has combined the latest servicing stack update (SSU) with your operating system’s latest cumulative update (LCU).

Download the Out-Of-Band Updates Manually

Follow the below steps to download the out-of-band updates manually.

• Open the browser and access the Microsoft Update Catalog.
• Paste the KB article number you want to download in the search box.

Click on the Download button to download the update.

You will get a new window opened once you click on the Download button. Click on .msu link and download the update locally on your machine. You can also see the UpdateID that would help you when you import the update into WSUS

Fix for Windows Server Crashes – Import Into WSUS

The above-listed updates will not be in the WSUS and SCCM database until you Import the Out-of-Band Updates Into WSUS. In his article, Kannan has already explained the best way to Import any updates into the WSUS and SCCM databases.

• How to Manually Import Patches to WSUS and Deploy using SCCM.
• Patch Missing from SCCM How to Import into WSUS Manually

The Import Updates feature in WSUS has undergone a replacement process to ensure its compatibility with current technology standards. The previous implementation depended on ActiveX technology, which has since been deprecated. Consequently, the Import Updates functionality has been revamped, and a more modern approach has been adopted. The new approach utilizes a PowerShell script, which has proven to be more efficient and effective for this purpose.

• Copy the below PowerShell script on the WSUS server and save it as ImportUpdateToWSUS.ps1 Thanks to Microsoft for the script

 <#
.SYNOPSIS
Powershell script to import an update, or multiple updates into WSUS based on the UpdateID from the catalog.

.DESCRIPTION
This script takes user input and attempts to connect to the WSUS server.
Then it tries to import the update using the provided UpdateID from the catalog.

.INPUTS
The script takes WSUS server Name/IP, WSUS server port, SSL configuration option and UpdateID as input. UpdateID can be viewed and copied from the update details page for any update in the catalog, https://catalog.update.microsoft.com. 

.OUTPUTS
Writes logging information to standard output.

.EXAMPLE
# Use with remote server IP, port and SSL
.\ImportUpdateToWSUS.ps1 -WsusServer 127.0.0.1 -PortNumber 8531 -UseSsl -UpdateId 12345678-90ab-cdef-1234-567890abcdef

.EXAMPLE
# Use with remote server Name, port and SSL
.\ImportUpdateToWSUS.ps1 -WsusServer WSUSServer1.us.contoso.com -PortNumber 8531 -UseSsl -UpdateId 12345678-90ab-cdef-1234-567890abcdef

.EXAMPLE
# Use with remote server IP, defaultport and no SSL
.\ImportUpdateToWSUS.ps1 -WsusServer 127.0.0.1  -UpdateId 12345678-90ab-cdef-1234-567890abcdef

.EXAMPLE
# Use with localhost default port
.\ImportUpdateToWSUS.ps1 -UpdateId 12345678-90ab-cdef-1234-567890abcdef

.EXAMPLE
# Use with localhost default port, file with updateID's
.\ImportUpdateToWSUS.ps1 -UpdateIdFilePath .\file.txt


.NOTES  
# On error, try enabling TLS: https://learn.microsoft.com/mem/configmgr/core/plan-design/security/enable-tls-1-2-client

# Sample registry add for the WSUS server from command line. Restarts the WSUSService and IIS after adding:
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 /V SchUseStrongCrypto /T REG_DWORD /D 1

## Sample registry add for the WSUS server from PowerShell. Restarts WSUSService and IIS after adding:
$registryPath = "HKLM:\Software\Microsoft\.NETFramework\v4.0.30319"
$Name = "SchUseStrongCrypto"
$value = "1" 
if (!(Test-Path $registryPath)) {
    New-Item -Path $registryPath -Force | Out-Null
}
New-ItemProperty -Path $registryPath -Name $name -Value $value -PropertyType DWORD -Force | Out-Null
Restart-Service WsusService, w3svc

# Update import logs/errors are under %ProgramFiles%\Update Services\LogFiles\SoftwareDistribution.log

#>

param(
    [Parameter(Mandatory = $false, HelpMessage = "Specifies the name of a WSUS server, if not specified connects to localhost")]
    # Specifies the name of a WSUS server, if not specified connects to localhost.
    [string]$WsusServer,

    [Parameter(Mandatory = $false, HelpMessage = "Specifies the port number to use to communicate with the upstream WSUS server, default is 8530")]
    # Specifies the port number to use to communicate with the upstream WSUS server, default is 8530.
    [ValidateSet("80", "443", "8530", "8531")]
    [int32]$PortNumber = 8530,

    [Parameter(Mandatory = $false, HelpMessage = "Specifies that the WSUS server should use Secure Sockets Layer (SSL) via HTTPS to communicate with an upstream server")]
    # Specifies that the WSUS server should use Secure Sockets Layer (SSL) via HTTPS to communicate with an upstream server.  
    [Switch]$UseSsl,

    [Parameter(Mandatory = $true, HelpMessage = "Specifies the update Id we should import to WSUS", ParameterSetName = "Single")]
    # Specifies the update Id we should import to WSUS
    [ValidateNotNullOrEmpty()]
    [String]$UpdateId,

    [Parameter(Mandatory = $true, HelpMessage = "Specifies path to a text file containing a list of update ID's on each line", ParameterSetName = "Multiple")]
    # Specifies path to a text file containing a list of update ID's on each line.
    [ValidateNotNullOrEmpty()]
    [String]$UpdateIdFilePath
)

Set-StrictMode -Version Latest

# set server options
$serverOptions = "Get-WsusServer"
if ($psBoundParameters.containsKey('WsusServer')) { $serverOptions += " -Name $WsusServer -PortNumber $PortNumber" }
if ($UseSsl) { $serverOptions += " -UseSsl" }

# empty updateID list
$updateList = @()

# get update id's
if ($UpdateIdFilePath) {
    if (Test-Path $UpdateIdFilePath) {
        foreach ($id in (Get-Content $UpdateIdFilePath)) {
            $updateList += $id.Trim()
        }
    }
    else {
        Write-Error "[$UpdateIdFilePath]: File not found"
		return
    }
}
else {
    $updateList = @($UpdateId)
}

# get WSUS server
Try {
    Write-Host "Attempting WSUS Connection using $serverOptions... " -NoNewline
    $server = invoke-expression $serverOptions
    Write-Host "Connection Successful"
}
Catch {
    Write-Error $_
    return
}

# empty file list
$FileList = @()

# call ImportUpdateFromCatalogSite on WSUS
foreach ($uid in $updateList) {
    Try {
        Write-Host "Attempting WSUS update import for Update ID: $uid... " -NoNewline
        $server.ImportUpdateFromCatalogSite($uid, $FileList)
        Write-Host "Import Successful"
    }
    Catch {
        Write-Error "Failed. $_"
    }
}

• Follow the above steps to download the updates manually and Copy the UpdateID.
• To Import the latest updates, please launch a PowerShell console with administrator privileges and execute the script using the provided syntax.

.\ImportUpdateToWSUS.ps1 -UpdateId 1bc3fe3f-74dc-48da-a3f3-65a6c06f4335

Open the WSUS Console, Expand the Updates tab. Select All Updates and click the Search icon in the right-hand panel.

Enter the KB article ID, which is recently imported, and click Find Now

Well, Updates are available in the WSUS console now. We’re all set to synchronize updates from WSUS with the SCCM database

SCCM – Manually Start Software Update Synchronization

Let’s check how to Synchronize WSUS with the SCCM database. To complete this work, follow the steps mentioned below.

• Open the SCCM Console
• Select the Software Library
• Expand Software Updates
• Select All Software Updates and right-click and select Synchronization Software updates

Open the WSUSSYnc.log from the Site server; you can find the imported update information. Then, you can create a new SUG for imported Out-of-Band Updates and deploy it on affected domain controllers.

Resources

• March 22, 2024—KB5037422 (OS Build 20348.2342) Out-of-band
• March 22, 2024—KB5037423 (OS Build 14393.6799) Out-of-band
• KB5037426: Update to address a known issue that affects LSASS in Windows Server 2012 R2
• WSUS and the Microsoft Update Catalog | Microsoft Learn

We trust that this article will greatly benefit you and your organization. Thank you for your patience in reading this post. I look forward to seeing you in the next post. Keep supporting the HTMD Community.

We are on WhatsApp. To get the latest step-by-step guides and news updates, Join our Channel. Click here – HTMD WhatsApp.

Author

About Author – Sujin Nelladath has over 10 years of experience in SCCM device management and Automation solutions. He writes and shares his experiences with Microsoft device management technologies, Azure, and PowerShell automation.