Welcome to Part 5 of the series Intune PKI Made Easy With Joy.
Today’s post is all about the different Intune SCEP HTTP errors that we may get while working with SCEP certificate deployments from Intune – how to identify their probable causes to help quicker troubleshooting to ensure lower downtime.
Disclaimer: The Intune SCEP HTTP Error causes as listed in this article are purely based on experience dealing with production environment, and may not match the order in which MS has listed the same in its troubleshooting document.
Previous articles of this series Intune PKI Made Easy With Joy
- Part 1 – Learn The Basic Concepts of PKI
- Part 2 – Knowing SCEP – The General Workflow
- Part 3 – Intune SCEP PKI Implementation Deep Dive
- Part 4 – Intune SCEP Certificate Workflow Analysis
Let’s get started.
Demystifying Intune SCEP HTTP Errors
In order for an internet-facing device to send the SCEP request to NDES, the request must go via a proxy. In most setups, Azure AD App Proxy (Microsoft recommended) exposes the internal NDES mscep.dll URL.
So let’s begin with the HTTP errors that we may likely get due to Azure AD App Proxy.
Intune SCEP HTTP Errors – AAD App Proxy Errors
504 Gateway Timeout
This mostly occurs if the AAD App Proxy connector is not in a Running state or the Server which hosts the connector has gone offline.
Startup Type of Microsoft AAD Application Proxy Connector service by default is set to Automatic (Delayed Start).
As such if the server recently went through a reboot cycle, ensure to check if the connector service has started or not.
502 Bad Gateway
This mostly occurs due to a typo in the internal URL while creating/configuring the App Proxy application in Azure, as a result of which the connector fails to resolve the internal URL resulting in the error.
This page cannot be displayed/Can’t reach this page
Occurs due to incorrect App Proxy configuration for publishing the SCEP URL.
If you choose to use your custom verified domain in Azure as the mscep URL domain rather than the default “msappproxy.net” domain, you would require to have a CNAME record created in the DNS provider for the custom domain to map the requests.
By default, Azure App Proxy requires no additional configuration for using the default “-.msappproxy.net” domain. Also, if you choose to use a custom domain other than the default “-msappproxy.net” domain, you would need to provide an SSL cert.
In this case, I went with the default config to save myself from the extra work but still got the error. Generating a Diagnostics
for the App Proxy application in Azure tells the error source.
While configuring the App Proxy application, I left the Pre Authentication set to Azure Active Directory.
For NDES, Pre Authentication setting within the AAD App Proxy application in Azure needs to be set to Passthrough since no actual AAD user will be accessing the application.
Provided that we are good from the proxy side and AAD App Proxy is able to resolve the external requests to the internal NDES MSCEP URL, the HTTP errors we will encounter from now onwards are all dependent on the physical infrastructure settings (NDES and PKI).
Intune SCEP HTTP Errors – Configuration Issues
Intune SCEP HTTP Error 500 – Internal Server Error
There can be several causes for this particular error and you would need to check several things to narrow down to the real cause.
NDES Service Account permission/rights issue
The SCEP pool in IIS runs under the NDES service account identity.
You need to check and ensure that the NDES Service Account
- is not in a Locked state,
- password is not Expired,
- is a member of the local IIS_IURS group on the NDES box (ensure there is no GPO that is modifying the membership of the particular group),
- the IIS_IURS group is assigned the Impersonate a client after authentication user right (by default the right is present unless modified),
- has Read permission to the MSCEP RA certs private key
If the above conditions are not satisfied, you will have the Intune SCEP HTTP Error 500 – Internal Server Error.
MSCEP RA Certificates are Expired (or Deleted, or Revoked)
NDES service startup depends on the MSCEP RA Certificates. [More of the NDES service startup sequence is discussed later in detail in this post]
If the certificates are in an expired state, were deleted, or revoked from the issuing CA for any causes, the NDES service will fail to start resulting in the Intune SCEP HTTP Error 500 – Internal Server Error.
CRP in IIS has Windows Authentication set to Enabled
Ensure CertificateRegistrationSvc in IIS [CRP] has Windows Authentication set to Disabled.
If it is set to Enabled as shown in the above snapshot, it may result in Intune SCEP HTTP Error 500 – Internal Server Error.
Note! This particular setting may also result in HTTP Error 503 – Service Unavailable.
Just like /certsrv/mscep app in SCEP pool, IIS authentication for CRP also needs to have
- Anonymous Authentication = Enabled
- Windows Authentication = Disabled
Issuing CA is Offline or Unreachable from NDES
*********One of the most notable causes of Intune SCEP HTTP Error 500 – Internal Server Error.*********
Network configuration changes done by the network/firewall/proxy team may result in the Issuing CA becoming unreachable or unavailable from the NDES box resulting in the error. Error events can be viewed from the events below.
Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService Event ID 8 - The Network Device Enrollment Service cannot retrieve information about the certification authority (0x80004005). Unspecified error Event ID 2 - The Network Device Enrollment Service cannot be started (0x80004005). Unspecified error
You can use certutil
command on the NDES server to verify CA availability. You should expect an output as below if your Issuing CA is not reachable or unavailable from the NDES box.
PS C:\Windows\system32> certutil -config - -ping WIN-2CQQDB9STE6.joymalya.xyz\joymalya-WIN-2CQQDB9STE6-CA Connecting to WIN-2CQQDB9STE6.joymalya.xyz\joymalya-WIN-2CQQDB9STE6-CA ... Server could not be reached: The RPC server is unavailable. 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) -- (32ms) CertUtil: -ping command FAILED: 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) CertUtil: The RPC server is unavailable.
The same can be verified using another tool named pkiview.msc
Sometimes, for a multi-tier PKI setup, it happens that Issuing CA Server is online but Certificate Services of Issuing CA fails to start post a server restart [legit causes] due to Root CA CRL has expired.
Consider the below classic example where when I do a normal ping, my CA server returns a response that means it’s online and available, but a certutil ping to the same CA server says otherwise.
PS C:\Windows\system32> ping WIN-2CQQDB9STE6.joymalya.xyz Pinging WIN-2CQQDB9STE6.joymalya.xyz [10.0.0.20] with 32 bytes of data: Reply from 10.0.0.20: bytes=32 time<1ms TTL=128 Reply from 10.0.0.20: bytes=32 time=1ms TTL=128 Reply from 10.0.0.20: bytes=32 time=1ms TTL=128 Reply from 10.0.0.20: bytes=32 time=1ms TTL=128 Ping statistics for 10.0.0.20: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 1ms, Average = 0ms PS C:\Windows\system32> certutil -ping WIN-2CQQDB9STE6.joymalya.xyz Connecting to WIN-2CQQDB9STE6.joymalya.xyz ... Server could not be reached: The RPC server is unavailable. 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) -- (32ms) CertUtil: -ping command FAILED: 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) CertUtil: The RPC server is unavailable.
This can happen for multi-tier PKI infra where the Root CA is mostly offline and Root CA CRL has expired. The Issuing CA went through a reboot due to any causes. Post reboot, the Certificate Services of Issuing CA will fail to start as when it tries to retrieve Root CA CRL to verify its own cert chain, the verify operation fails.
Fixing the issue might require engagement from both the Network team and/or ADCS team as this does not generally fall within the Intune scope.
NDES unable to retrieve CRL from CDP
******This is the other most prominent cause of Intune SCEP HTTP Error 500 – Internal Server Error******
This occurs due to the NDES box failing to retrieve the Certificate Revocation List (CRL from CDP) which is required for the NDES service to start itself.
Remember that NDES is implemented as an ISAPI extension in IIS, as such you will not see NDES as a service when you check-in services.msc. Instead, the service runs within the w3wp.exe
a process which is an IIS worker process.
Now for the SCEP pool in IIS to run, the NDES service must first start successfully. You can get from here the NDES service startup checklist.
In a nutshell, NDES Service Startup Sequence is –
- Locate RA cert from machine cert store – X509Objects
- Acquire the Private Keys – CryptAcquireCertificatePrivateKey
- Build cert chain – CertGetCertificateChainStart
- Verify Revocation Status – CertVerifyRevocationStart
If all the steps complete successfully, the NDES service starts and you get the success events as below
Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService Event ID 47 - The Network Device Enrollment Service loaded the Registration Authority (RA) key exchange certificate with serial number ###### from the "MY" store. Event ID 48 - The Network Device Enrollment Service loaded the Registration Authority (RA) signature certificate with serial number ###### from the "MY" store. Event ID 1 - The Network Device Enrollment Service started successfully.
However, the NDES service fails to start if it fails to verify the MSCEP-RA certificates. Error events are as below.
Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService Event ID 8 - The Network Device Enrollment Service cannot retrieve information about the certification authority (0x80004005). Unspecified error Event ID 2 - The Network Device Enrollment Service cannot be started (0x80004005). Unspecified error
The event as above does not necessarily means that the MSCEP RA certs are expired or not present (deleted), but can also be because of Expired CRL or NDES server unable to retrieve the CRL, though the events as above do not specifically state such.
To check the actual reason for failure to load the RA certs, you need to enable CAPI2 logging in events [Applications and Services logs > Microsoft > Windows > CAPI2], then restart the Cryptographic Services and IIS [PS command iisreset] and check back in the CAPI2 Operational logs.
CAPI 2 Event Trace for same is shown below
Event ID 90 - X509 Objects - Loads RA cert Event ID 70 - Acquire Certificate Private Key - Crypt Function CryptAcquireCertificatePrivateKey Event ID 10 - Build Chain - Crypt Function CertGetCertificateChainStart to start Chain Building Event ID 40 - Verify Revocation - Crypt Function CertVerifyRevocationStart to start revocation status Event ID 52 - Retrieve Object from Network - Crypt Function CryptRetrieveObjectByUrlWireStart to retrieve CRL
If CA CRL has expired or the CRL location is offline, the error events start from here
Error Event ID 53 - Retrieve Object from Network - Crypt Function CryptRetrieveObjectByUrlWire which will report error if CRL not retrievable. Error Event ID 42 - Reject Revocation Information - Crypt Function CertRejectedRevocationInfo. Here you can see that if CRL was actually retrieved, the failure reason might be due to Action CheckTimeValidity [retrieved CRL has expired] Error Event ID 41 - Verify Revocation - Crypt Function CertVerifyRevocation will give you the outcome of revocation verification.
In the NDES server, you can use certutil command certutil -urlcache CRL
to view the locally cached CRL URLs and then use certutil command certutil -URL <LDAP or HTTP URL of CRL>
to verify if the NDES server is able to retrieve the CRL from CDP.
You will see the URL Retrieval Tool GUI window like this. Select CRLs (from CDP) and click on Retrieve. In an ideal scenario, the status comes as OK.
Important! If the communication between servers is facilitated by a proxy server, ensure that proxy settings in all the servers involved are configured for both user and machine/system context.
In the production environment, there may be a firewall in between or there may be a proxy that is facilitating the communications. For a firewall, you need to ensure you have the proper exceptions set to allow connections. For proxy, you need to set proxy settings in the SYSTEM context for NDES to work. The best way is to use the PSEXEC tool.
Open a CMD using PSEXEC and confirm the CMD process is running as SYSTEM using the command whoami. Once confirmed, you can then use the netsh winhttp show proxy to view if the proxy settings are correctly configured as SYSTEM context or if not, use the netsh winhttp to set proxy the command to set the same.
Fixing issues with retrieving CRL or expired CRL mostly should fall under ADCS scope and not Intune scope in general.
Once the CRL issue is resolved, on your NDES server, restart the Cryptographic Services and IIS and wait to check if NDES services started successfully. [Event ID 1 for the successful startup of NDES]
Intune SCEP Http Error 503 – Service Unavailable
This one is pretty straightforward in most cases you will find the IIS SCEP pool has crashed (stopped stated). The likely causes are
Untrusted Intermediate Certificates in NDES server cert store
This is very common for multi-tier PKI infrastructure. Hence, if you are facing this issue, the first thing you should check on your NDES box is to open PowerShell and run the following command
Get-Childitem -Path cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}
If the above code gives any output, means there are untrusted certs present and you can simply delete them by running the below command
Get-Childitem -Path cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}| Remove-Object
Note! If you face this issue frequently, consider creating a scheduled task to run this command automatically at regular intervals to get rid of the untrusted certificates.
- IIS SSL binding certificate was renewed but the NDES Policy module still points to the old SSL certificate thumbprint
Remember the certificate used to bind port 443 of IIS is the same that needs to be selected while installing the Intune NDES certificate connector. As such, this situation arises mostly when the IIS SSL Binding Certificate gets renewed but the same is not updated with the Intune NDES Certificate Connector.
You must ensure that the thumbprint of the certificate held by the NDES Policy Module must match the thumbprint of the SSL binding certificate used in IIS.
Intune NDES Connector service requires to access the URLs in CRL for proper functioning. As such, post successful NDES service startup, if for any reason, the CRL URLs become unreachable again from the NDES box, may result in HTTP Error 503 – Service Unavailable.
Notice that this time, the NDES service itself has started, otherwise, the same scenario during NDES service startup results in HTTP Error 500 – Internal Server Error.
The End
In addition to all the Intune SCEP HTTP Errors as listed above, you may also encounter HTTP Error 414 – Request URI Too Long and HTTP Error 413 – Payload Too Large which are all due to incorrect IIS Request Filtering settings for the /certsrv/mscep
virtual app.
The causes listed for the errors presented in this article may not be exhaustive and while troubleshooting you may encounter a completely new cause which is quite possible. I would be very much interested to learn if you have discovered any such new cause for an error listed above while troubleshooting in your environment.
Well, that was all in store for today. Will be back with some other posts shortly. Till then, continue to stay safe…
Good stuff can you do a blog on HA for NDES? I do not see much content on this subject.
HA for NDES is also on my radar. Hopefully, it will be sooner than later.
Hi, any hint if we encounter event id 29 error in Network Device enrollment ? thanks.
Hi Sebastien,
Though an old post, but have you checked https://docs.microsoft.com/en-us/archive/blogs/tune_in_to_windows_intune/ndes-event-id-29-the-password-in-the-certificate-request-cannot-be-verified
Hello Joy,
I also ran into HTTP 500 error when setting up NDES server, logged a case with MS and didn’t revert back for 2 weeks and then I came across your blog which helped me to look at the CRL location and turns out in our case the CRL location was wrongly published. thanks and keep sharing!!
Hi Surjeet,
Great to hear that the blog helped you.
We have a subject name: false error on our IIS cert. Any tips?
Great extensive post!
I’m still confused about the relation with a third party on-prem forward proxy and Azure Application (reverse) proxy.
I cannot get an Internet connection through our forward on-prem proxy when signing in to Azure via the Intune certificate connector.
However, if I browse to that URL via IE on my NDES server to https://portal.manage.microsoft.com, I get an HTTP 403 ‘forbidden’. This makes sense since we have to authenticate through our on-prem proxy with an account.
But after authenticating and configuring this proxy in the Intune Connector configuration, I still don’t get access.
Any help would be appreciated, thanks!
NDES Disable Password Requirement.
I’ve read a few blogs and articles that say;
“There is no way for Cisco devices to supply the required password to enroll with NDES/MSCEP, so you need to disable the requirement for a password.”
This is NOT TRUE, however the whole point of issuing certificates via your PKI infrastructure, is that it can scale dramatically. If you are creating passwords and embedding those passwords in all your enrollments, it can get a little unwieldy. So it may be sensible to remove the password requirement.
1. Windows Key+R > regedit {Enter} > Navigate to;
HKEY_LOCAL_MACHINE > SOFTWARE > Microsoft > Cryptography > MSCEP > EnforcePassword > EnforcePassword
To disable change the value to 0 (zero).
Disable NDES Password Enforce
Update: 22/10/21: You may also need to recycle the SCEP application pool in IIS (on the Certificate Services Server)
From IIS Manager > CA > Application Pools, SCEP. > From the right hand panel > Advanced Settings. > Set Load User Profile to ‘True‘ > OK.
Again in the right panel > Recycle > From IIS Manager > Sites > Default Web Site. > From the right panel, click Restart.
Wonderful article
Any chance you have an updated screen shot for CRP in IIS has Windows Authentication set to Enabled . It looks like it doesnt create this website anymore with the new connector