Import Keycloak existing realm without losing existing users

Question

I configured the kubernetes init container that imports an existing Realm, and override the one that is in environment already.

I'm using this command:

/opt/keycloak/bin/kc.sh import --file=/opt/keycloak/data/import/tyk-realm-export.json

The problem that I'm having, is, when the existing realm is replaced, it deletes all users in it.

Is there any way to import a new configuration for realm without loosing the users? In particular, my DB is expecting to have hundred thousands of users.

PS: using keycloak >=18.0.0

Here is a log:

Appending additional Java properties to JAVA_OPTS: -Dkeycloak.profile.feature.upload_scripts=enabled -Dkeycloak.migration.strategy=OVERWRITE_EXISTING
2022-06-17 10:17:30,048 INFO  [org.keycloak.common.Profile] (main) Preview feature enabled: scripts
2022-06-17 10:17:30,198 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: <MyHostname>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: true
2022-06-17 10:17:32,225 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-06-17 10:17:32,505 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-06-17 10:17:32,559 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-06-17 10:17:33,004 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.9.Final
2022-06-17 10:17:33,311 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2022-06-17 10:17:33,312 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2022-06-17 10:17:33,599 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-06-17 10:17:33,600 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20.00MB, but the OS only allocated 212.99KB
2022-06-17 10:17:33,600 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-06-17 10:17:33,600 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25.00MB, but the OS only allocated 212.99KB
2022-06-17 10:17:35,614 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) sb-keycloak-bd4778849-n8jh5-3122: no members discovered after 2004 ms: creating cluster as coordinator
2022-06-17 10:17:35,636 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [sb-keycloak-bd4778849-n8jh5-3122|0] (1) [sb-keycloak-bd4778849-n8jh5-3122]
2022-06-17 10:17:35,647 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `sb-keycloak-bd4778849-n8jh5-3122`, physical addresses are `[10.2.0.74:41912]`
2022-06-17 10:17:36,678 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: sb-keycloak-bd4778849-n8jh5-3122, Site name: null
2022-06-17 10:17:37,972 INFO  [org.keycloak.services] (main) KC-SERVICES0030: Full model import requested. Strategy: OVERWRITE_EXISTING
2022-06-17 10:17:37,983 INFO  [org.keycloak.exportimport.singlefile.SingleFileImportProvider] (main) Full importing from file /opt/keycloak/data/import/tyk-realm-export.json
2022-06-17 10:17:38,388 INFO  [org.keycloak.exportimport.util.ImportUtils] (main) Realm 'tyk' already exists. Removing it before import
2022-06-17 10:17:49,348 INFO  [org.keycloak.exportimport.util.ImportUtils] (main) Realm 'tyk' imported
2022-06-17 10:17:49,540 INFO  [org.keycloak.services] (main) KC-SERVICES0032: Import finished successfully
2022-06-17 10:17:49,832 INFO  [io.quarkus] (main) Keycloak 18.0.1 on JVM (powered by Quarkus 2.7.5.Final) started in 25.524s. Listening on: http://0.0.0.0:8080
2022-06-17 10:17:49,834 INFO  [io.quarkus] (main) Profile import_export activated. 
2022-06-17 10:17:49,834 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]
2022-06-17 10:17:49,922 INFO  [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel `ISPN`
2022-06-17 10:17:50,012 INFO  [io.quarkus] (main) Keycloak stopped in 0.165s
Done

Deepdyed

Have you found a solution for this problem? – Skye

No. Still no solution for this – Deepdyed

Answer 1

I don't know your exact use-case.

But the question I ask: is it mandatory to import the realm again or do you just need an update?

First time you import the realm, it's perfectly fine. When importing you have to choose between two strategies: OVERWRITE_EXISTING and IGNORE_EXISTING .

However, both don't fit the use-case of updating particular items of your realm, like the smtp-server settings.

Let's say you have three environments: development, release, production.

Your configuration evolves and runs through each stage.

With ignore_existing no import will happen.

With overwrite_existing it will remove all your users, since overwrite_existing works this way: delete the existing, completly create a new realm. No need to say that this is not wanted in a productive environment.

What you need in this case is just an update via the REST-API. (note that this links points to a specific version AND PLEASE NOTE THAT THE SPECIFIED PATH IN THE DOCUMENTATION IS WRONG, THAT'S WHY IT DIFFERS IN MY CURL COMMAND)

E.g.: Let's say you get the requirement, that the emails sent by keycloak should have a new "from"-mail. You develop it, it will be tested and than run in production. In this case you can run cUrl-scripts like this:

------------------------------
# First initialize your variables
export KEYCLOAK_HOST="http://localhost:8471"
export REALM_NAME="myrealm"
export CLIENT_SECRET="client-secret-from-your-admin-cli-user-in-the-myrealm"
export CLIENT_ID="admin-cli"


# get the token (mandatory for any action as an admin)
export TOKEN=$( \
        curl -s \
        -d "client_id=$CLIENT_ID" \
        -d "client_secret=$CLIENT_SECRET" \
        -d 'grant_type=client_credentials' \
        "$KEYCLOAK_HOST/auth/realms/$REALM_NAME/protocol/openid-connect/token" \
        | jq -j '.access_token')
        
#update your specific resource, in this case we're updating the attribute smtpServer with the according values
curl -X PUT \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"  \
    -d '{"smtpServer" : { "replyToDisplayName" : "my Example Display Name", "starttls" : "false", "auth" : "", "port" : "12345", "host" : "my-host.local", "replyTo" : "[email protected]", "from" : "[email protected]", "fromDisplayName" : "", "ssl" : ""} }' \
    $KEYCLOAK_HOST/auth/admin/realms/ekc 

With this approach you can update your realm and let it evolve according to its stage.

As I said, I don't know if it solves your problem, but if so I am happy I could help.

Skye

Thank you, I'll check it when I need a update a next time. – Deepdyed

Answer 2

Maybe you could export both realms and stitch the dumps together.

Antakiya

It's only one realm ("tyk"). OPs problem is importing a realm which already exists with Strategy: OVERWRITE_EXISTING. What keycloak does at this point: delete the existing realm and create it again instead of an update/merge mechanism. The consequence is, that all users that were created before are now deleted. – Skye