Importing realm 8.0.2 (Docker)

[wecler]

Hello there,
after an export and several attempts to import my realm to another docker container I decided to write here a post and ask for help.

I used to have a docker container with configured Keycloak realm which I was using. I exported my configuration and wanted to create a new one. (Docker)

I tried it with docker-compose so in my environment configuration I’ve added

KEYCLOAK_IMPORT=path/to/file.json

and ran it up. I ended with an error:

Unable to import realm test from file.

My first thoughts were that I did something wrong so I looked up at the docs (on hub.docker) and realised that there is a way to do it manually by standalone.sh. I also tried it with no success.

INFO  [org.keycloak.exportimport.dir.DirImportProvider] (ServerService Thread Pool -- 70) Importing     from directory /opt/jboss/keycloak/imports/keycloak-export/test-realm.json
ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 70) MSC000001: Failed to start service jboss.deployment.unit."keycloak-server.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit
."keycloak-server.war".undertow-deployment: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()

+

    ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "keycloak-server.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.d
eployment.unit.\"keycloak-server.war\".undertow-deployment" => "java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()

I tried also this:

• https://github.com/keycloak/keycloak-documentation/blob/master/server_admin/topics/export-import.adoc

without any success.

Also when I got my fresh keycloak installation with master realm, import in panel does not work for my export. It does work when I create a new realm with name of the realm I would like to import and import it in there.

What also should be mentioned that while importing an existing realm the console says that it’s about to delete the existing one and afterwards fails.

Has somebody faced similar issue and got a fix for this?

Thanks for help in advance.
Greetings.

[jangaraj]

Cant import realm using Docker Image

I am using the following docker image docker run -e KEYCLOAK_USER=demo -e KEYCLOAK_PASSWORD=demo -e KEYCLOAK_IMPORT=/tmp/quarkus-realm.json -v /tmp/quarkus-realm.json:/tmp/quarkus-realm.json -p 8180:8080 quay.io/keycloak/keycloak The realm doesnt show up in the realms on the UI. I also tried to import it from the UI and gives the following exception: 00:51:37,894 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-7) Uncaught server error: java.lang.RuntimeException:…

[Robinyo]

See: Cant import realm using Docker Image

[wecler]

Thank you @jangaraj and @Robinyo for your answers.

I assume you both meant the last comment from @Robinyo.

I saw it before posting mine, well I tried it today again and it’s not working.

What I did:

1. started a container with keycloak (my previous one)
2. entered my container with keycloak (docker exec)
3. exported data from keycloak to keycloak-export.json (standalone)
4. copied json from container to my local directory (docker cp)
5. added volume in my docker-compose to keycloak
6. started my second container with keycloak (docker-compose)
7. entered my fresh created container with keycloak (docker exec)
8. executed import method as described in the given thread (standalone)

Errors:

[org.keycloak.exportimport.util.ImportUtils] (ServerService Thread Pool – 59) Realm ‘master’ already exists. Removing it before import
ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool – 59) MSC000001: Failed to start service jboss.deployment.unit.“keycloak-server.war”.undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit.“keycloak-server.war”.undertow-deployment: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()

ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation (“add”) failed - address: ([(“deployment” => “keycloak-server.war”)]) - failure description: {“WFLYCTL0080: Failed services” => {“jboss.deployment.unit."keycloak-server.war".undertow-deployment” => “java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.NullPointerException”}}
INFO [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested via an OS signal

ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation (“add”) failed - address: ([(“subsystem” => “microprofile-metrics-smallrye”)]): java.util.ConcurrentModificationException

INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0004: Undertow 2.0.27.Final stopping
INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0028: Stopped deployment keycloak-server.war (runtime-name: keycloak-server.war) in 209ms
ERROR [org.jboss.as.server] (ServerService Thread Pool – 43) WFLYSRV0021: Deploy of deployment “keycloak-server.war” was rolled back with the following failure message:
{“WFLYCTL0080: Failed services” => {“jboss.deployment.unit."keycloak-server.war".undertow-deployment” => “java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.NullPointerException”}}

I’ll include my docker-compose in case that I did something something wrong here:

version: '3'
services:
  keycloak:
    image: jboss/keycloak
    environment:
      KEYCLOAK_USER: admin
      KEYCLOAK_PASSWORD: admin
      DB_VENDOR: h2
      DB_USER: keycloak
      DB_PASSWORD: keycloak
      DB_DATABASE: keycloak
      DB_ADDR: h2
    ports:
      - 8080:8080
    volumes:
    - ./imports/:/tmp

Any further ideas what could resolve my issue?
Thanks for help!

[Robinyo]

@wecler

What I did …

You need to provide the commands that you executed (rather than just describing what you think you did). The following steps (with the actual commands) worked for me.

When you start your Keycloak container mount a volume:

docker run -d --name keycloak \
  -p 10001:8080 \
  -v ~/workspace/Robinyo/serendipity:/serendipity \
  -e KEYCLOAK_USER=admin \
  -e KEYCLOAK_PASSWORD=secret \
  jboss/keycloak

Export

Then export your database into a single JSON file:

docker exec -it keycloak /opt/jboss/keycloak/bin/standalone.sh \
  -Djboss.socket.binding.port-offset=100 \
  -Dkeycloak.migration.action=export \
  -Dkeycloak.migration.provider=singleFile \
  -Dkeycloak.migration.file=/serendipity/keycloak-export.json

Import from the Command Line

Start with a blank canvas:

docker container stop keycloak
docker container rm keycloak

docker run -d --name keycloak \
  -p 10001:8080 \
  -v ~/workspace/Robinyo/serendipity:/serendipity \
  -e KEYCLOAK_USER=admin \
  -e KEYCLOAK_PASSWORD=secret \
  jboss/keycloak

To import from a (previously exported) file into your database:

docker exec -it keycloak /opt/jboss/keycloak/bin/standalone.sh \
  -Djboss.socket.binding.port-offset=100 \
  -Dkeycloak.migration.action=import \
  -Dkeycloak.migration.provider=singleFile \
  -Dkeycloak.migration.file=/serendipity/keycloak-export.json

When the import is complete use Ctrl-C to exit the session.

[wecler]

Well you’re right. So I did following (with commands executed)

1. Started my container

docker-compose up

2. Entered my container

docker exec -it {CONTAINER} bash

3. switched the directory to /opt/jboss/keycloak/bin

cd /opt/jboss/keycloak/bin

4. executed export command

./standalone.sh
-Djboss.socket.binding.port-offset=100
-Dkeycloak.migration.action=export
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.file=/tmp/keycloak-export.json

5. left container

exit

6. copied key-cloak export from container to my pc

docker cp container:/tmp/keycloak-export.json .

7. shut my containers down

docker-compose down

8. mounted the file to my new container

docker-compose file:
keycloak:
image: jboss/keycloak
environment:
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: admin
DB_VENDOR: h2
DB_USER: keycloak
DB_PASSWORD: keycloak
DB_DATABASE: keycloak
DB_ADDR: h2
expose:
- 8080
volumes:
- ./imports/:/tmp

and ran it up with

docker-compose up

9. entered my container with

docker exec -it {CONTAINER} bash

10. checked if my export file has been sucessfuly mounted

ls /tmp/

11. switched to keycloak dir

cd /opt/jboss/keycloak/bin

12. ran import command

./standalone.sh
-Djboss.socket.binding.port-offset=100
-Dkeycloak.migration.action=import
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.file=/tmp/keycloak-export.json

step 12 fails with following error

afer that

Caused by: java.lang.NullPointerException

and

07:11:40,986 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation (“add”) failed - address: ([(“deployment” => “keycloak-server.war”)]) - failure description: {“WFLYCTL0080: Failed services” => {“jboss.deployment.un
it."keycloak-server.war".undertow-deployment” => “java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.NullPointerException”}}

Thank you for your help.

[erickmoreno]

@wecler

./standalone.sh
-Djboss.socket.binding.port-offset=100
-Dkeycloak.migration.action=export
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.file=/tmp/keycloak-export.json

I think that I’m facing the same issue, but I noticed that the problem can be at the export step, instead of the import.

Can you verify if the file generated by the export command really contains the realm that you’re looking for?

In my case, the command that you used to export your data only exports de master realm and in a invalid format. When I tried to export only the relevant realm. I used this command:

./standalone.sh -Dkeycloak.migration.action=export 
-Dkeycloak.migration.provider=singleFile 
-Dkeycloak.migration.realmName=realm-name 
-Dkeycloak.migration.usersExportStrategy=REALM_FILE 
-Dkeycloak.migration.file=realm-name-realm.json

And got an exception:

ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "microprofile-metrics-smallrye")]): java.lang.NullPointerException
	at org.wildfly.extension.microprofile.metrics-smallrye@18.0.1.Final//org.wildfly.extension.microprofile.metrics.MicroProfileMetricsSubsystemAdd$2.execute(MicroProfileMetricsSubsystemAdd.java:86)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1413)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:521)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:472)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:434)
	at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.ServerService.boot(ServerService.java:435)
	at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.ServerService.boot(ServerService.java:394)
	at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:374)
	at java.base/java.lang.Thread.run(Thread.java:834)

Can you verify if you’re not facing the same issue?

[wecler]

@erickmoreno I’ve looked at my export and it does contain my two realms. Master and my Test realm.

I think that your exception is being thrown because of missing port binding. Try adding

-Djboss.socket.binding.port-offset=100

You should notice that nothing happens and your file is not being exported because there was no action specified.

-Dkeycloak.migration.action=export

after adding these two lines to your execution command the file is successfully exported.
Full command:

./standalone.sh
-Djboss.socket.binding.port-offset=100
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.realmName=test
-Dkeycloak.migration.usersExportStrategy=REALM_FILE
-Dkeycloak.migration.action=export
-Dkeycloak.migration.file=/tmp/test.json

of course in the exported file only one realm exists. (the test one)

Afterwards I tried to import it with

./standalone.sh
-Djboss.socket.binding.port-offset=100
-Dkeycloak.migration.action=import
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.file=/tmp/test.json

it still does not work and throws the same errors as already posted.

What is also a weird behaviour the exported file can not be imported in the web interface. It only works when I manually create realm with the same name and then import the exported data.

[zonaut]

Maybe the problem is something else entirely.
When trying to do the same as you, importing a single JSON file containing the master realm and other I get the following fatal error earlier than the one you mentioned.

FATAL [org.keycloak.services] (ServerService Thread Pool – 70) java.lang.RuntimeException: Failed to parse json

The export JSON file is valid but maybe Keycloak can’t import a single file with multiple realms?
I am running 9.0.0.

UPDATE Import/export multiple realms which contains master

The following works for me, I needed to add - Dkeycloak.profile.feature.upload_scripts=enabled
Keycloak uses it’s own API to upload the files and then import them

FILE

Export

-Dkeycloak.migration.action=export \
-Dkeycloak.migration.provider=singleFile \
-Dkeycloak.migration.file=/opt/jboss/keycloak/realm-export.json \
-Dkeycloak.migration.strategy=OVERWRITE_EXISTING

Import

-Dkeycloak.migration.action=import \
-Dkeycloak.migration.provider=singleFile \
-Dkeycloak.profile.feature.upload_scripts=enabled \
-Dkeycloak.migration.file=/opt/jboss/keycloak/realm-import.json \
-Dkeycloak.migration.strategy=OVERWRITE_EXISTING

DIR

But now I decided to use the dir export/import strategy as this makes the files more readable and manageable.

Export

-Dkeycloak.migration.action=export \
-Dkeycloak.migration.provider=dir \
-Dkeycloak.migration.dir=/opt/jboss/keycloak/export-dir \
-Dkeycloak.migration.usersPerFile=1000 \
-Dkeycloak.migration.strategy=OVERWRITE_EXISTING

Import

-Dkeycloak.migration.action=import \
-Dkeycloak.migration.provider=dir \
-Dkeycloak.profile.feature.upload_scripts=enabled \
-Dkeycloak.migration.dir=/opt/jboss/keycloak/import-dir \
-Dkeycloak.migration.strategy=OVERWRITE_EXISTING

[wecler]

@zonaut, thank you for your answer.

I’ve executed your commands and for me there’s also missing:

-Djboss.socket.binding.port-offset=100

because of

Failed to start service org.wildfly.undertow.listener.default: org.jboss.msc.service.StartException in service org.wildfly.undertow.listener.default: Address already in use /127.0.0.1:8080

The export action was sucessfull as always - both methods tested singleFile and dir.
There was no problem with json format in my case.

and of course the import fails…

Command executed:

./standalone.sh
-Djboss.socket.binding.port-offset=100
-Dkeycloak.migration.action=import
-Dkeycloak.migration.provider=dir
-Dkeycloak.profile.feature.upload_scripts=enabled
-Dkeycloak.migration.dir=/opt/jboss/keycloak/imports/export-dir
-Dkeycloak.migration.strategy=OVERWRITE_EXISTING

Errors:

07:11:37,341 INFO [org.keycloak.exportimport.dir.DirImportProvider] (ServerService Thread Pool – 63) Importing from directory /opt/jboss/keycloak/imports/export-dir
07:11:37,831 INFO [org.keycloak.services] (ServerService Thread Pool – 63) KC-SERVICES0030: Full model import requested. Strategy: OVERWRITE_EXISTING
07:11:38,100 INFO [org.keycloak.exportimport.util.ImportUtils] (ServerService Thread Pool – 63) Realm ‘master’ already exists. Removing it before import
07:11:40,395 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool – 63) MSC000001: Failed to start service jboss.deployment.unit.“keycloak-server.war”.undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit.“keycloak-server.war”.undertow-deployment: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
at org.wildfly.extension.undertow@18.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)

Caused by: java.lang.NullPointerException
at org.keycloak.keycloak-services@8.0.2//org.keycloak.url.DefaultHostnameProvider.resolveUri(DefaultHostnameProvider.java:83)

07:11:40,440 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation (“add”) failed - address: ([(“deployment” => “keycloak-server.war”)]) - failure description: {“WFLYCTL0080: Failed services” => {“jboss.deployment.unit."keycloak-server.war".undertow-deployment” => “java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication()
Caused by: java.lang.NullPointerException”}}

07:11:40,513 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation (“add”) failed - address: ([(“subsystem” => “microprofile-metrics-smallrye”)]): java.lang.NullPointerException
at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.descriptions.DefaultResourceDescriptionProvider.getModelDescription(DefaultResourceDescriptionProvider.java:177)

Warning: it’s huge!

Keycloak - Importing realm 8.0.2 error - Pastebin.com

I could sadly not provide the full error message here because of:

Body is limited to 32000 characters; you entered 53720.

[zonaut]

Are you by any chance trying to upgrade for an older version.
According to the release notes and upgrade notes there was a change in the hostname provider.
https://www.keycloak.org/docs/latest/upgrading/#new-default-hostname-provider
Just throwing some ideas here now, check you standalone-ha.xml for spi name=“hostname” and see if that complies with the new ones if you are upgrading.

It seems this is a bug in 8.0.2

KEYCLOAK-12190 Add validation for client root and base URLs

https://issues.redhat.com/browse/KEYCLOAK-12921

https://issues.redhat.com/browse/KEYCLOAK-12921?jql=project%20%3D%20KEYCLOAK%20AND%20affectedVersion%20%3D%208.0.2%20AND%20text%20~%20"import"%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

Try 8.0.1 or use 9.0.0 or the fix mentioned in https://issues.redhat.com/browse/KEYCLOAK-12921

[wecler]

@zonaut, that was a clean installation from a docker image.
I searched the web for something similar, even went through jira tickets but haven’t found anything about it. (I must be blind, because you found something :D)

I’ve retried with 9.0.0 and it works.

Thank you a lot.

[zonaut]

It was a search on org.keycloak.url.DefaultHostnameProvider.resolveUri(DefaultHostnameProvider that brought me there, I wouldn’t have found it otherwise

[wlad]

It seems not to work as documented here

I’ve tried …

docker run --rm --name keycloak \
-p 8081:8080 \
-e KEYCLOAK_USER=admin \
-e KEYCLOAK_PASSWORD=admin \
-e KEYCLOAK_IMPORT=$(pwd)/preconfigured_realms/ehrbase-realm.json -v $(pwd)/preconfigured_realms/ehrbase-realm.json:/tmp/ehrbase-realm.json \
-it quay.io/keycloak/keycloak:10.0.2 \

and …

docker run --rm --name keycloak \
-p 8081:8080 \
-e KEYCLOAK_USER=admin \
-e KEYCLOAK_PASSWORD=admin \
-e KEYCLOAK_IMPORT=$(pwd)/preconfigured_realms/ehrbase-realm.json -v $(pwd)/preconfigured_realms/ehrbase-realm.json:/tmp/ehrbase-realm.json \
-it quay.io/keycloak/keycloak:10.0.2 \
-Dkeycloak.profile.feature.upload_scripts=enabled

resulting in error

11:38:48,752 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "microprofile-metrics-smallrye")]): java.lang.NullPointerException
    at org.wildfly.extension.microprofile.metrics-smallrye@19.1.0.Final//org.wildfly.extension.microprofile.metrics.MicroProfileMetricsSubsystemAdd$2.execute(MicroProfileMetricsSubsystemAdd.java:86)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1413)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:527)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:515)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:477)
    at org.jboss.as.server@11.1.1.Final//org.jboss.as.server.ServerService.boot(ServerService.java:448)
    at org.jboss.as.server@11.1.1.Final//org.jboss.as.server.ServerService.boot(ServerService.java:401)
    at org.jboss.as.controller@11.1.1.Final//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:416)
    at java.base/java.lang.Thread.run(Thread.java:834)

[cchantep]

I confirm that even with the latest 12.0.4 Docker image, the realm import using env var REALM_IMPORT is no only not working but prevent Keycloak from starting.

[dankobgd]

Yeah it just doesn’t work in 12.0.4 here as well.
Export works fine but import doesn’t work at all, and yes i tried everything i read on the whole internet and i still get Java NullPointerException