Scaling an IdentityServer4 service

Question

I have followed the IdentityServer4 quickstarts and am able to authenticate my javascript web page (almost the same as that provided in the quickstart) with a localhosted instance of IdentityServer using the Implicit grant. Again, my IdentityServer is almost exactly the same as that provided in the quickstart mentioned above - it just has some custom user details.

I then moved my application (C# .NET Core) into a docker container and have hosted one instance of this within a Kubernetes cluster (single instance) and created a Kubernetes service (facade over one or more 'real' services) which lets me access the identity server from outside the cluster. I can modify my JavaScript web page and point it at my Kubernetes service and it will still quite happily show the login page and it seems to work as expected.

When I then scale the IdentityServer to three instances (all served behind a single Kubernetes service), I start running into problems. The Kubernetes service round-robins requests to each identity server, so the first will display the login page, but the second will try and handle the authentication after I press the login button. This results in the following error:

System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {19742e88-9dc6-44a0-9e89-e7b09db83329} was not found in the key ring. at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData) at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken) --- End of inner exception stack trace --- ... And lots more......

So - I understand that I am getting this error because the expectation is that the same IdentityServer should service the request for a page that it has shown (otherwise how would the anti-forgery token work, right?), but what I am trying to understand is how I can make this work in a replicated environment.

I don't want to host multiple identity servers on different IP's/ports; I'm trying to build a HA configuration where if one IdentityServer dies, nothing calling the endpoint should care (because requests should be serviced by other working instances).

I said i was using the quickstart code - this means that in the startup of the IdentityServer, there is code that looks like this...

    public void ConfigureServices(IServiceCollection services)  
    {
        services.AddMvc();

        services.AddIdentityServer(options =>
            {
                options.Events.RaiseSuccessEvents = true;
                options.Events.RaiseFailureEvents = true;
                options.Events.RaiseErrorEvents = true;
            })
            .AddTemporarySigningCredential()
            .AddInMemoryIdentityResources(Config.GetIdentityResources())
            .AddInMemoryApiResources(Config.GetApiResources())
            .AddInMemoryClients(Config.GetClients())

I am assuming that I need to replace the .AddTemporarySigningCredential() logic with a certificate that can be used by all instances of the IdentityServer which are running in my Kubernetes cluster. Not knowing how MVC really works (MVC6 is used to generate the login pages in the IdentityServer service, which I got from the example code - link above) - I want to know if just changing the code to use a proper certificate which is shared between all services will be sufficient to get a prototype HA IdentityServer cluster working?

By working, I mean that my expectation is that I can have n number of IdentityServer instances running in a Kubernetes cluster, have a Kubernetes service to act as a facade over however many IdentityServer's I have running, and be able to authenticate using multiple IdentityServer instances which can share data to the extent that they all provide exactly the same authority to my calling web applications, and can handle each other's requests in the event that one or more instances die.

Any help or insight would be appreciated.

Bellyache

Add TemporarySingingCrendentials will pot. generate a different value for each pod. you will definitly have to a static certificate . see .AddSigningCredential But i'm not sure if this will be the single issue. as i plan to do something very similar thats quite interessting – Awestricken

We are using Identity Server across a load balanced environment without problems, but we're not using a flow that involves a log-in page generated by IdSvr. We are using a single certificate for signing (not the temp one) and that is straightforward to set-up/test so you should be able to determine whether your problem is that temp cert or something else pretty quickly. – Ruthenic

@Ruthenic that it's good to know; I was having a bit of a problem with something else earlier so haven't had a chance to try the .AddSigningCredential() as suggested by Boas yet. I will update when i get the certificate updated - probably after the weekend now. – Bellyache

Why was this post down-voted? It'd be nice if whoever did that could leave a note to say what they think is wrong. – Bellyache

Answer 1

I think I have worked this out. To resolve my issue, I have done two things:

1. Create my own X509 certificate and shared this certificate between each of my IdentityServer's. There are lots of examples of how to create valid certificates on the net; I just used

   services.AddIdentityServer(...).AddSigningCredential(new X509Certificate2(bytes, "password")
   

   in my startup class.

2. Dug into the MVC framework code and worked out that I needed to implement a Key storage provider in order to share state between different instances of the MVC part of Identity Server which serves up the login page.

It turns out that there is a Redis backed KSP available from NuGet, which means that I just need to spin up a private redis instance in my Kube cluster (which isn't accessible outside of my cluster) to share decryption secrets.

/* Note: Use an IP, or resolve from DNS prior to adding redis based key store as direct DNS resolution doesn't work for this inside a K8s cluster, though it works quite happily in a Windows environment. */  
var redis = ConnectionMultiplexer.Connect("1.2.3.4:6379");
services.AddDataProtection()
        .PersistKeysToRedis(redis, "DataProtection-Keys");

I can now scale my identity service to 3 instances and have a Kube service acting as a facade over all the available instances. I can watch the logs as Kubernetes round-robin's requests between the identity service, and my authentication happens just as I expect.

Thanks to those who commented on the question prior to this post.

Bellyache

I also done these two things, but I still have this issue. – Globin

Hey @jay Quick question. What happens if the Redis server where you have your keys stored gets wiped? Do you have to re-login in all Identity Server instances, or does the application automatically generate new keys that can be used with the old cookie? Thank you! – Kozhikode

@HubertJarema - I cat say I've ever noticed having to do anything other than log-in again. I've wiped my Redis database hundreds of times since this post, and I've never been given any cause to give it any thought :) – Bellyache

Answer 2

For those using Kubernetes, it's possible to use the File System Key Storage Provider

public void ConfigureServices(IServiceCollection services)
{
    services.AddDataProtection()
        .PersistKeysToFileSystem(new DirectoryInfo(@"/app/key-storage"));
}

where the directory '/app/key-storage' is mapped to an nfs backed persistent volume.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-key-storage
spec:
  selector:
    matchLabels:
      type: nfs-pv
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Mi

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
  labels:
    type: nfs-pv
spec:
  storageClassName: manual
  capacity:
    storage: 10Mi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <server>
    path: /<path>
  persistentVolumeReclaimPolicy: Delete

And in the IDP Deployment

template:
  spec:
    containers:
      - name: <name>
        volumeMounts:
          - name: key-storage
            mountPath: /app/key-storage
            readOnly: false
    volumes:
      - name: key-storage
        persistentVolumeClaim:
        claimName: pvc-key-storage

And you need the signing cert. This can added as a secret, and then the IDP Deployment can use another volume to mount the secret (not shown).

apiVersion: v1
kind: Secret
metadata:
  name: cert-secret
  labels:
    app: <app-label>
type: Opaque
data:
  signingcert.pfx: <base64 cert value>

Illustrious

Hey I'm trying to achieve exactly this (using a shared folder across the kubernetes pods) and still having problems with dataprotection. Is there any other configuration involved? Thanks – Declared

Answer 3

1. make the configuration and operational data stores persistent.
2. use a x509 cert to signing the key and for validation.
3. add dataprotection and keep the keys shared (in redis, filestore or blobstorage).

let me know if you need the code blocks for these.

Castaway