.NET Core X509Certificate2 usage (under Windows/IIS, Docker, Linux)

Question

I am really trying a long time to use certificates in .NET Core API.

Basically where I need to use them is in a .NET Core web api running on IIS and docker.

Certificates I need to use are for:

Microsoft.AspNetCore.DataProtection

public void ConfigureServices(IServiceCollection services)
{
  services.AddDataProtection()
    .PersistKeysToFileSystem(new DirectoryInfo(dataProtectionKeystorePath))
    .ProtectKeysWithCertificate
    (
      new X509Certificate2(dataProtectionCertificatePath, dataProtectionCertificateSecret)
    );
}

Kestrel Options to set SSL certificate

public static IWebHost BuildWebHost(string[] args) =>
  WebHost.CreateDefaultBuilder(args)
    .UseKestrel
    (
      options =>
        {
          options.Listen(address, port, listenOptions =>
            {
              listenOptions.UseHttps(new X509Certificate2(sslCertificatePath, sslCertifciateSecret));
            }
          );
        }
      )
      // other setup
     ;

IdentityServer4.SigningCredentials

Note: this code works on the dev machine starting from VS2017 but throws those exceptions on Windows 2008 R2 IIS test server.

services.AddIdentityServer()
  .AddSigningCredential
  (
    new X509Certificate2(tokenCertificatePath, tokenCertificatePassphrase)
  )
  // other setup
  ;

All three imply place a certificate file, load it using the constructor, pass the secret and let's go.

sarkasm > Happy me, it's so easy. < sarkasm

So I created a certs subdir holding the certifciates. Added settings for the secrets. Verified all values are loaded/created as intended. Verified the files are at the intended locations and therefor exist. In short something like:

string dataProtectionKeystorePath = System.Path.Combine(Environment.ContentRootPath, "keystore");
string dataProtectionCertificatePath = System.Path.Combine(Environment.ContentRootPath, "certs", "keystore.pfx");
string dataProtectionSecret = Configuration.GetSection("CertificateSecrets").GetValue<string>("keystoreSecret", null);

string tokenCertificatePath = System.Path.Combine(Environment.ContentRootPath, "certs", "token.pfx");
string tokenCertificateSecret = Configuration.GetSection("CertificateSecrets").GetValue<string>("tokenSecret", null);

string sslCertificatePath = System.Path.Combine(Environment.ContentRootPath, "certs", "ssl.pfx");
string sslCertificateSecret = Configuration.GetSection("CertificateSecrets").GetValue<string>("tokenSecret", null);

Let's have fun they said. So let's go and deploy.

What I end up with are those exceptions:

Data Protection Exception:

info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[58]
      Creating key {39560783-e349-475e-8e3f-748abb8c6c8b} with creation date 2018-11-16 08:01:49Z, activation date 2018-11-16 08:01:49Z, and expiration date 2019-02-14 08:01:49Z.
info: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[39]
      Writing data to file '[intentionally removed for post]'.
fail: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[24]
      An exception occurred while processing the key element '<key id="39560783-e349-475e-8e3f-748abb8c6c8b" version="1" />'.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__61_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.GetKeyFromCert(EncryptedKey encryptedKey, KeyInfoX509Data keyInfo)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.DecryptEncryptedKey(EncryptedKey encryptedKey)
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver[12]
      Key {39560783-e349-475e-8e3f-748abb8c6c8b} is ineligible to be the default key because its CreateEncryptor method failed.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__61_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.GetKeyFromCert(EncryptedKey encryptedKey, KeyInfoX509Data keyInfo)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.DecryptEncryptedKey(EncryptedKey encryptedKey)
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor()
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key)
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver[12]
      Key {39560783-e349-475e-8e3f-748abb8c6c8b} is ineligible to be the default key because its CreateEncryptor method failed.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__61_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.GetKeyFromCert(EncryptedKey encryptedKey, KeyInfoX509Data keyInfo)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.DecryptEncryptedKey(EncryptedKey encryptedKey)
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
   at System.Lazy`1.CreateValue()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor()
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key)

The exception for the Identity Server 4 Signing Credentials

Application startup exception: Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: An internal error occurred
   at Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(Byte[] rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags)
   at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
   at AuthServer.Startup.ConfigureServices(IServiceCollection services) in [intentionally removed for post]\Startup.cs:line 136
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
crit: Microsoft.AspNetCore.Hosting.Internal.WebHost[6]
      Application startup exception
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: An internal error occurred
   at Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(Byte[] rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags)
   at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
   at AuthServer.Startup.ConfigureServices(IServiceCollection services) in [intentionally removed for post]\Startup.cs:line 136
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()

The exception for the SSL certificate

Unhandled Exception: Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: An internal error occurred
   at Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(Byte[] rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags)
   at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
   at AuthServer.Program.<>c.<BuildWebHost>b__1_3(ListenOptions listenOptions) in [intentionally removed for post]\Program.cs:line 58
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions.Listen(IPEndPoint endPoint, Action`1 configure)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions.Listen(IPAddress address, Int32 port, Action`1 configure)
   at Microsoft.Extensions.Options.ConfigureNamedOptions`1.Configure(String name, TOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsManager`1.<>c__DisplayClass5_0.<Get>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
   at Microsoft.Extensions.Options.OptionsManager`1.Get(String name)
   at Microsoft.Extensions.Options.OptionsManager`1.get_Value()
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.CreateServiceContext(IOptions`1 options, ILoggerFactory loggerFactory)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer..ctor(IOptions`1 options, ITransportFactory transportFactory, ILoggerFactory loggerFactory)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(SingletonCallSite singletonCallSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureServer()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.StartAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String shutdownMessage)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host)
   at MyProject.Program.Main(String[] args) in [intentionally removed for post]\Program.cs:line 47

Facing this certificate exceptions for months now, googling a lot and facing this exceptions again whatever I tried (according to google results, questions asked on github etc..) it comes to my mind that I miss something import.

I tried to use StorageFlags but didn't found one (combination) working.

The projects should run on Windows (IIS), docker and maybe one time linux (ubuntu, debian). That's why we decided to place certs in a (mounted accross multiple instances) subdir. So the posts I found suggesting installing the certificates into the microsoft certificates store will not work. How could it on docker and linux?

Since all certificates I tend to use are affected I came to the point that the problem could be me having a major missunderstanding in how to use the certificates.

Can anyone help me figuring out the major point i missed and finally get the certificates to run? Do I need to configure something? If so what?

---

Just some investigation summary:

Am I sure that the files exist: yes because otherwise the exception would be

System.Security.Cryptography.CryptographicException: System cannot find specified file.

Am I sure that the password is correct: yes because otherwise the exception would be

Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: The specified network password is not correct.

File permissions: Thanks to CheshireCat reminding me. On the Windows Server 2008 R2 IIS 7.5 I checked the file permissions to the certs subdir for the user DefaultAppPool (thought it would be a different user unitl now). Full access file permissions to the certs subdir were granted to the user DefaultAppPool like shown in this link.

Usually the IS4 signing credentials exceptions occur on Application StartUp. Using the X509KeyStorageFlags.MachineKeySet does not throw on start up but shows the login window and throws after login. No token is returned but a session is created so the user can modify the ASP NET Identity account settings.

Certificate expiration: certificates can expire. Data protection seems not to check for the expiration date so certificates can be used even after they expired. Identity Server 4 signing credentials are still can be used but ASP.NET core token validation will raise unauthorized if certifciate expires.

---

Problem solution

After one and a half year the solution is: a somehow broken (self-signed/self-created) certificate.

If the certificate works can be checked with this code:

  static void Main(string[] args)
  {
    X509Certificate2 cert = new X509Certificate2(Path.Combine(Directory.GetCurrentDirectory(), "cert.pfx"), "password");
    Console.WriteLine("cert private key: " + cert.PrivateKey);
  }

If you see the following output the certificate is good, otherwise you get exceptions as described above.

cert private key: System.Security.Cryptography.RSACng

Creating certificates for development

SSL certificate

Use the dotnet tool dev-certs. This call opens a prompt and then imports the localhost SSL certificate into CurrentUser certificate store (view it with MMC). No code changes in Program needed.

dotnet dev-certs https --trust

Other certifcates

The "broken" certs were created using makecert. I switched to OpenSSL but also could be New-SelfSignedCertificate tool used (if you are using Win 10).

If OpenSSL is installed correctly and added to the PATH variable the following batch file creates working certificates (user interaction required!).

  openssl genrsa 2048 > private.pem
  openssl req -x509 -days 365 -new -key private.pem -out public.pem
  openssl pkcs12 -export -in public.pem -inkey private.pem -out cert.pfx
  REM openssl pkcs12 -info -in cert.pfx
  certutil -dump cert.pfx
  PAUSE

Setup for the local develompment (Win 7, VS 2017, IIS Express/Kestrel)

As Cheshire-Cat (again a big thank you) described the code for all parts work as expected under .NET Core 2.1. Including data-protection and Identity Server 4 signing credentials.

---

Deployment to IIS 7.5 on Windows 2008 R2

The fun continues. Getting the certificates to work on the local development machine and deploying it to IIS raises "new" exceptions.

• checked file permissions on IIS (DefaultAppPool has full rights)
• certifcates are the same that worked on the local machine
• no certificates used are installed (not on the development machine, not on the IIS machine) so the actual files should be taken.

logs show a basically good setup

  using data protection keystore path C:\inetpub\wwwroot\auth\keystore
  data protection keystore is protected with certificate at path 'C:\inetpub\wwwroot\auth\certs\keystore.pfx'
  loading keystore certificate from file 'C:\inetpub\wwwroot\auth\certs\keystore.pfx'
  loading keystore certificate with passphrase ********

  Application startup exception: Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: An internal error occurred
     at Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(Byte[] rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags)
     at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
     at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
     at [SomeOfMyNamespaces].X509Certificate2Loader.LoadFromFile(String certName, String certPath, String certPassphrase, ILogger logger) in [intentionally-blanked]\X509Certificate2Loader.cs:line 57
     at [SomeOfMyNamespaces].DataProtectionExtensions.AddDataProtection(IServiceCollection services, IConfiguration config, IHostingEnvironment environment, String applicationName, String applicationVersion, ILogger logger) in [intentionally-blanked]\DataProtectionExtensions.cs:line 207
     at [SomeOfMyNamespaces].Startup.ConfigureServices(IServiceCollection services) in [intentionally-blanked]\Startup.cs:line 96
  --- End of stack trace from previous location where exception was thrown ---
     at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
     at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
     at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
  --- End of stack trace from previous location where exception was thrown ---
     at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
  crit: Microsoft.AspNetCore.Hosting.Internal.WebHost[6]
        Application startup exception
  Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: An internal error occurred
     at Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(Byte[] rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags)
     at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
     at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
     at [SomeOfMyNamespaces].X509Certificate2Loader.LoadFromFile(String certName, String certPath, String certPassphrase, ILogger logger) in [intentionally-blanked]\X509Certificate2Loader.cs:line 57
     at [SomeOfMyNamespaces].DataProtectionExtensions.AddDataProtection(IServiceCollection services, IConfiguration config, IHostingEnvironment environment, String applicationName, String applicationVersion, ILogger logger) in [intentionally-blanked]\DataProtectionExtensions.cs:line 207
     at [SomeOfMyNamespaces].Startup.ConfigureServices(IServiceCollection services) in [intentionally-blanked]\Startup.cs:line 96
  --- End of stack trace from previous location where exception was thrown ---
     at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
     at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
     at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
  --- End of stack trace from previous location where exception was thrown ---
     at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()

The referenced line 57 is:

  cert = new X509Certificate2(certPath, certPassphrase);

The same exception occurs for the Identity Server 4 signing credentials.

Giving another bounty of 100 for the answer helping me get this running on IIS on Windows Server 2008 R2.

Solution for IIS and docker (and local development too): using this and neither IIS nor Docker setup will complain:

cert = new X509Certificate2(certPath, certPassphrase, X509KeyStorageFlags.MachineKeySet);

Specialty

Remember that if you are running multiple instances that the keys have to be stored in the same place so that all instances can access the keys. – Lifegiving

Yes, thx for the reminder. That's the reason why we mount them (under docker) as a sub dir to the app directory. – Specialty

I tried that to. Kept getting an error that they were not secure so it wouldnt use them. I tried to get this to work for a few months myself gave up and use Redis for data protection now. – Lifegiving

Some results I found during my resarch hold the answer "thats was asked so many times, so let's not answer". Seems to be more difficult than expected. There also seems to be a bug that will be fixed in .NET Core 2.2 that may affect my/our data protection problem at least. – Specialty

Thats the issue I found but I am not sure if that is the problem: github.com/aspnet/AspNetCore/issues/2321 – Specialty

Well now that explains a lot – Lifegiving

Do you think this could cause all 3 problems? – Specialty

Ok, after re-reading the issues about data protection it seems that should work with .NET Core 2.1 but obviously it isn't. – Specialty

Please also take a look at this. Maybe it can help. – Cresol

Did you try specifying the UserKeySet flag? If that doesn't solve it: how did you create the certificates? – Contango

@Contango Certificates were created using makecert and pvk2pfx. Sure I need to mind things for the SSL certificate but is there anything to mind creating certs for dataprotection and signing credentials? – Specialty

What? They promised me that I can start 3 simultaneous bounties.... Wanted to start a bounty of 50 for each setup... but can't start the other two bounties.... – Specialty

Answer 1

My actual code running on my IS4 project is this:

X509Certificate2 certificate = null;

// Load certificate from Certificate Store using the configured Thumbprint
using (X509Store store = new X509Store(StoreName.My, StoreLocation.LocalMachine))
{
    store.Open(OpenFlags.ReadOnly);
    X509Certificate2Collection certificates = store.Certificates.Find(X509FindType.FindByThumbprint, appConfiguration.CertificateThumbprint, false);

    if (certificates.Count > 0)
        certificate = certificates[0];
}

// Fallback to load certificate from local file
if (certificate == null)
{
    string path = Path.Combine("C:\\Certificates", appConfiguration.CertificateFilename);

    try
    {
        certificate = new X509Certificate2(path, "CertificateSecret123$");
        logger.LogInformation($"Found from file {certificate.Thumbprint}");
    }
    catch (Exception ex)
    {
        logger.LogError(ex, $"Certificate file error {path}");
        certificate = null;
    }
}

if (certificate == null)
    throw new Exception($"Certificate {appConfiguration.CertificateThumbprint} not found.");

builder.AddSigningCredential(certificate);

If I change the thumbprint to force the code looking for certificate from local file, I get the expected result:

UPDATE 2018/11/19

I corrected a bug in the code due to the fact that in case the certificate file is not found, the X509Certificate2 class constructor throws an exception. So, in fact, the previous if (certificate == null) control after the line certificate = new X509Certificate2..., would never have been reached.

Steps to make code works are:

1. Create certificate and put it in a directory
2. Check directory permissions on the folder for the user that run the code.
   • If you're on development machine, check that the user that runs VS has permissions to reach the directory where the file is placed.
   • If you're on IIS, check the AppPool that hosts your application and the user with whom it is executed. It needs to has permissions on the certificate directory.
3. Load certificate from file using the sample code. Be sure of the correctness of the path.
4. Load certificate from Store using the thumbprint. To get the certificate thumbprint you can:
   • Use PowerShell command Get-ChildItem -path cert:\LocalMachine\My specifing the correct place in the Store where you have registered the certificate.
   • Use Windows Explorer by right click on .pfx file -> Open -> locate your file inside the MMC console -> double click on it -> Details -> Thumbprint. IMPORTANT NOTE: there's a well known ugly bug in Windows, when coping/pasting the Thumbprint value from the preview textbox. A HIDDEN character is added at the beginnig of the thumbprint when you paste the value. So copy/paste it first on a text editor and then check it by moving with arrow keys near the first character. See also this.

Cresol

To be honest I don't know if that would/wouldn't work on docker. If you say it would work it would be an option. – Specialty

Thats not adding the data protection though. Which means its still not used for (reset password, user conformation) token generation. – Lifegiving

In order to get this working do I need to install the certificate somehow or is it enough to have it in my certs directory? – Specialty

I'm using the same code in my PC and in DEV environment. On my PC I've no certificate registered, so I always fallback to just picking up the certificate file from the path and it works. It has nothing to do with the installed certificates, since the code is only looking for a file on a specified path. – Cresol

@Specialty Please also consider file permissions! The user that runs your code, in this case I guess it is ApplicationPoolIdentity (if you didn't changed it), needs to have access to the certificate path. Make a check. – Cresol

@CheshireCat "Funny" thing is that the signing credentials do work (with the code posted by me in the question) on my development machine but throw the exception on IIS. I tried it with full access to the DefaultAppPool user setup. Same internal error. For the log how to set it up: learn.microsoft.com/en-us/iis/manage/configuring-security/… – Specialty

But the hint with the file permissions was good. Always thought it would be a different user. – Specialty

How do I figure out the thumbprint of a certificate? – Specialty

With PowerShell: Get-ChildItem -path cert:\LocalMachine\My. With Windows Explorer: right click on .pfx file -> Open -> locate your file in the MMC console -> double click on it -> Details -> Thumbprint. IMPORTANT NOTE: there's a well known ugly bug in Windows, when coping/pasting the Thumbprint value from the preview textbox. A HIDDEN character is added at the beginnig of the thumbprint when you paste the value. So copy/paste it first on a text editor and make a check. See also this – Cresol

I searched but didn't find a way to check file permissions .NET core. Do you know a way? – Specialty

Well, my problem is solved for the local development setup. I will try to reward you with the bounty and update the question. If you want to receive another bounty now I will have to set it up for our dev IIS. – Specialty

@Specialty Did you already updated the question? Anyway, what exactly are the issues you're addressing in the development IIS environment that you've not in your dev machine? Note: thanks for quoting me on the question and for the bounty! :-) – Cresol

@CheshireCat Sorry for the delay. Had to ensure everything is intended, make builds and deploy. Ready for a new run with a new bounty of 50 for the IIS setup? :-) – Specialty

Seems your code uses the default UserKeySet StorageFlag too. Did you managed to give the application access rights to the user store? It seems to work on IIS if I switch to MachineKeySet – Specialty

@Specialty I didn't give any access rights to my application. Anyway, on Microsoft Docs you can see that: "Private keys are stored in the current user store rather than the local computer store. This occurs even if the certificate specifies that the keys should go in the local computer store.", if it can helps you. – Cresol