Px-backup fails for OIDC

Hi Team,

I am trying to get px-backup installed on IBM IKS cluster without Portworx enterprise and I used the spec generator to generate the HELM chart. But when I deploy it I get this failure in the frontend pod

Warning Failed 2m1s (x8 over 3m51s) kubelet, 10.94.170.188 Error: couldn’t find key OIDC_CLIENT_SECRET in Secret test/pxc-backup-secret

I have tried using the “default” serviceaccount in a namespace, also tried creating new service account and used the ClientId and ClientSecret from the secret yaml. But it always fails. Please let me know if you need more details.

Warm Regards
Sandeep

I’m experiencing the same issue, any hint would be appreciated!

Hi guys,

Can you please provide the following outputs?

  1. helm list -n <installed_namespace>
  2. kubectl get pod -n <installed_namespace>
  3. Logs of pod "pxcentral-post-install-hook-"
    kubectl logs pxcentral-post-install-hook-
    -n <installed_namespace>
    In case of complete failure of the post-install job, you may see 5 pods of the above kind(pxcentral-post-install-hook-*), please provide the logs for the very first and second ones.
  4. Logs of keycloak pod
    kubectl logs pxcentral-keycloak-0 -n <installed_namespace>

Hi, thanks for your reply.

1 # helm list -n px-backup
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
px-backup px-backup 1 2021-01-08 09:32:25.863256801 +0000 UTC deployed px-backup-1.2.1 1.2.1

2 # oc get pod -n px-backup
NAME READY STATUS RESTARTS AGE
px-backup-5db9cf7f9d-89vzg 1/1 Running 2 5m6s
pxc-backup-etcd-0 1/1 Running 0 5m5s
pxc-backup-etcd-1 1/1 Running 0 5m5s
pxc-backup-etcd-2 1/1 Running 0 5m5s
pxcentral-apiserver-69bdc66697-9t7s7 1/1 Running 0 5m6s
pxcentral-backend-6d895b9446-jlgrs 0/1 Init:CreateContainerConfigError 0 5m6s
pxcentral-frontend-5c48787ddb-sr5j6 0/1 CreateContainerConfigError 0 5m6s
pxcentral-keycloak-0 1/1 Running 0 5m5s
pxcentral-keycloak-postgresql-0 1/1 Running 0 5m5s
pxcentral-lh-middleware-785959cd5f-6s6fl 1/1 Running 0 5m6s
pxcentral-mysql-0 1/1 Running 0 5m6s
pxcentral-post-install-hook-4zmfx 0/1 Error 0 2m17s
pxcentral-post-install-hook-75jxc 0/1 Error 0 3m4s
pxcentral-post-install-hook-8l8xd 0/1 Error 0 2m37s
pxcentral-post-install-hook-l54ms 0/1 Error 0 5m6s
pxcentral-post-install-hook-rzz8c 0/1 Error 0 97s

3 # oc logs pxcentral-post-install-hook-4zmfx -n px-backup
OIDC endpoint: pxcentral-keycloak-http:80
Statefulset name: [pxcentral-keycloak] Total replicas:[1], Ready replicas:[1]
Keycloak pod running status: [True]
Waiting for keycloak to start accepting connections…
Username: [admin], Email:[admin@portworx.com], Central OIDC enabled: [true], Central OIDC client ID: [pxcentral]
External OIDC enabled: [false], Client ID:[], Client Secret : [], Endpoint: []
Disable ssl command: kubectl exec -it pxcentral-keycloak-0 --namespace px-backup – bash -c “cd /opt/jboss/keycloak/bin/ && ./kcadm.sh config credentials --server http :// localhost:8080/auth --realm master --user ‘admin’ --password ‘admin’ && ./kcadm.sh update realms/master -s sslRequired=NONE”
Disable ssl command status: [1], output:[Unable to use a TTY - input is not a terminal or the right kind of file
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0
Logging into http://localhost:8080/auth as user admin of realm master
Failed to create config file: /.keycloak/kcadm.config
command terminated with exit code 1]
Keycloak ssl disable status: [False]
Failed to disable ssl at keycloak.
Access token fetch data: {‘grant_type’: ‘password’, ‘client_id’: ‘admin-cli’, ‘username’: ‘admin’, ‘password’: ‘admin’, ‘token-duration’: ‘365d’}
Keycloak token fetch endpoint: http : // pxcentral-keycloak-http:80/auth/realms/master/protocol/openid-connect/token
API response status code: 403
Step 1: Access token fetch : FAILED
Keycloak configure status: False
Keycloak configuration failed, status: False
restarting job…

4 # oc logs pxcentral-keycloak-0 -n px-backup
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0
Added ‘admin’ to ‘/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json’, restart server to load user
=========================================================================

  Using PostgreSQL database

=========================================================================

Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0
09:33:23,175 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
09:33:23,322 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
09:33:23,338 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
09:33:23,534 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) starting
09:33:23,669 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
09:33:24,556 INFO  [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.10.4.Final
09:33:25,535 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:25,632 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:25,836 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
09:33:25,845 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
09:33:25,941 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
09:33:25,943 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) started in 2752ms - Started 55 of 78 services (32 services are lazy, passive or on-demand)
The batch executed successfully
09:33:26,145 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) stopped in 27ms
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0
09:33:27,740 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
09:33:27,810 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
09:33:27,820 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
09:33:27,953 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) starting
09:33:28,121 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
09:33:29,107 INFO  [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.10.4.Final
09:33:30,143 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:30,237 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:30,380 INFO  [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
09:33:30,404 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-1) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
09:33:30,536 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
09:33:30,538 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) started in 2791ms - Started 55 of 85 services (39 services are lazy, passive or on-demand)
The batch executed successfully
09:33:30,749 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) stopped in 33ms
Setting JGroups discovery to dns.DNS_PING with properties {dns_query=>pxcentral-keycloak-headless}
Executing cli script: /opt/jboss/startup-scripts/keycloak.cli
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0
09:33:36,574 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
09:33:36,643 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
09:33:36,651 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
09:33:36,782 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) starting
09:33:36,922 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
09:33:37,754 INFO  [org.wildfly.security] (ServerService Thread Pool -- 17) ELY00001: WildFly Elytron version 1.10.4.Final
09:33:38,413 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:38,495 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:38,628 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
09:33:38,646 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-1) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
09:33:38,725 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
09:33:38,727 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) started in 2141ms - Started 55 of 78 services (32 services are lazy, passive or on-demand)
The batch executed successfully
09:33:38,886 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) stopped in 33ms
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/keycloak

  JAVA: java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0
09:33:39,750 INFO  [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final
09:33:40,252 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final
09:33:40,263 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
09:33:40,394 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) starting
09:33:40,531 INFO  [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
09:33:41,436 INFO  [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.10.4.Final
09:33:42,131 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:42,165 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 13) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
09:33:42,319 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
09:33:42,342 INFO  [org.xnio] (MSC service thread 1-1) XNIO version 3.7.3.Final
09:33:42,355 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.7.3.Final
09:33:42,393 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 37) WFLYCLINF0001: Activating Infinispan subsystem.
09:33:42,443 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 51) WFLYSEC0002: Activating Security Subsystem
09:33:42,446 INFO  [org.wildfly.extension.microprofile.config.smallrye._private] (ServerService Thread Pool -- 45) WFLYCONF0001: Activating WildFly MicroProfile Config Subsystem
09:33:42,454 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 48) WFLYNAM0001: Activating Naming Subsystem
09:33:42,454 INFO  [org.jboss.as.connector] (MSC service thread 1-1) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.4.17.Final)
09:33:42,488 INFO  [org.jboss.as.jaxrs] (ServerService Thread Pool -- 39) WFLYRS0016: RESTEasy version 3.9.1.Final
09:33:42,494 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 38) WFLYIO001: Worker 'default' has auto-configured to 2 core threads with 16 task threads based on your 1 available processors
09:33:42,503 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 5.0.15.Final
09:33:42,506 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 32) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4)
09:33:42,522 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0018: Started Driver service with driver-name = h2
09:33:42,538 INFO  [org.wildfly.extension.microprofile.health.smallrye] (ServerService Thread Pool -- 46) WFLYHEALTH0001: Activating Eclipse MicroProfile Health Subsystem
09:33:42,558 INFO  [org.wildfly.extension.microprofile.metrics.smallrye] (ServerService Thread Pool -- 47) WFLYMETRICS0001: Activating Eclipse MicroProfile Metrics Subsystem
09:33:42,571 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 32) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.postgresql.Driver (version 42.2)
09:33:42,605 INFO  [org.jboss.as.security] (MSC service thread 1-1) WFLYSEC0001: Current PicketBox version=5.0.3.Final
09:33:42,726 WARN  [org.wildfly.clustering.web.undertow] (ServerService Thread Pool -- 54) WFLYCLWEBUT0007: No routing provider found for default-server; using legacy provider based on static configuration
09:33:42,793 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = postgresql
09:33:42,893 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 2.0.27.Final starting
09:33:42,957 INFO  [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service
09:33:43,059 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 4 (per class), which is derived from the number of CPUs on this host.
09:33:43,059 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 16 (per class), which is derived from thread worker pool sizing.
09:33:43,058 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 54) WFLYUT0014: Creating file handler for path '/opt/jboss/keycloak/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
09:33:43,073 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
09:33:43,393 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0012: Started server default-server.
09:33:43,406 INFO  [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
09:33:43,592 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0018: Host default-host starting
09:33:43,622 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
09:33:43,642 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "keycloak-server.war" (runtime-name: "keycloak-server.war")
09:33:43,653 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/keycloak/standalone/deployments
09:33:43,675 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow HTTP listener default listening on 0.0.0.0:8080
09:33:43,798 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0493: EJB subsystem suspension complete
09:33:43,832 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow HTTPS listener https listening on 0.0.0.0:8443
09:33:43,904 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS]
09:33:43,904 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
09:33:44,418 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Infinity Minus ONE +2' 9.4.16.Final
09:33:44,812 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60) WFLYCLINF0002: Started authenticationSessions cache from keycloak container
09:33:44,814 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started loginFailures cache from keycloak container
09:33:44,818 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 58) WFLYCLINF0002: Started offlineClientSessions cache from keycloak container
09:33:44,819 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 56) WFLYCLINF0002: Started work cache from keycloak container
09:33:44,819 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 67) WFLYCLINF0002: Started actionTokens cache from keycloak container
09:33:44,820 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 61) WFLYCLINF0002: Started offlineSessions cache from keycloak container
09:33:44,820 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 59) WFLYCLINF0002: Started clientSessions cache from keycloak container
09:33:44,821 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started sessions cache from keycloak container
09:33:44,822 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 57) WFLYCLINF0002: Started authorization cache from keycloak container
09:33:44,822 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started realms cache from keycloak container
09:33:44,823 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started keys cache from keycloak container
09:33:44,824 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started users cache from keycloak container
09:33:44,931 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0002: Started client-mappings cache from ejb container
09:33:44,977 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0273: Excluded subsystem webservices via jboss-deployment-structure.xml does not exist.
09:33:45,673 INFO  [org.keycloak.services] (ServerService Thread Pool -- 68) KC-SERVICES0001: Loading config from standalone.xml or domain.xml
09:33:46,070 INFO  [org.keycloak.url.DefaultHostnameProviderFactory] (ServerService Thread Pool -- 68) Frontend: <request>, Admin: <frontend>, Backend: <request>
09:33:46,334 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0002: Started realmRevisions cache from keycloak container
09:33:46,339 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0002: Started userRevisions cache from keycloak container
09:33:46,348 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0002: Started authorizationRevisions cache from keycloak container
09:33:46,350 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (ServerService Thread Pool -- 68) Node name: pxcentral-keycloak-0, Site name: null
09:33:49,808 INFO  [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 68) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
09:34:02,360 INFO  [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 68) HHH000204: Processing PersistenceUnitInfo [
	name: keycloak-default
	...]
09:34:02,445 INFO  [org.hibernate.Version] (ServerService Thread Pool -- 68) HHH000412: Hibernate Core {5.3.13.Final}
09:34:02,447 INFO  [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 68) HHH000206: hibernate.properties not found
09:34:02,675 INFO  [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 68) HCANN000001: Hibernate Commons Annotations {5.0.5.Final}
09:34:02,873 INFO  [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 68) HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL95Dialect
09:34:03,587 INFO  [org.hibernate.engine.jdbc.env.internal.LobCreatorBuilderImpl] (ServerService Thread Pool -- 68) HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
09:34:03,592 INFO  [org.hibernate.type.BasicTypeRegistry] (ServerService Thread Pool -- 68) HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType@3aa77c03
09:34:03,598 INFO  [org.hibernate.envers.boot.internal.EnversServiceImpl] (ServerService Thread Pool -- 68) Envers integration enabled? : true
09:34:04,108 INFO  [org.hibernate.orm.beans] (ServerService Thread Pool -- 68) HHH10005002: No explicit CDI BeanManager reference was passed to Hibernate, but CDI is available on the Hibernate ClassLoader.
09:34:04,174 INFO  [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 68) HV000001: Hibernate Validator 6.0.18.Final
09:34:05,497 INFO  [org.hibernate.hql.internal.QueryTranslatorFactoryInitiator] (ServerService Thread Pool -- 68) HHH000397: Using ASTQueryTranslatorFactory
09:34:06,098 INFO  [org.keycloak.services] (ServerService Thread Pool -- 68) KC-SERVICES0050: Initializing master realm
09:34:09,028 INFO  [org.keycloak.services] (ServerService Thread Pool -- 68) KC-SERVICES0006: Importing users from '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json'
09:34:09,726 INFO  [org.keycloak.services] (ServerService Thread Pool -- 68) KC-SERVICES0009: Added user 'admin' to realm 'master'
09:34:09,905 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002225: Deploying javax.ws.rs.core.Application: class org.keycloak.services.resources.KeycloakApplication
09:34:09,906 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002205: Adding provider class org.keycloak.services.error.KeycloakErrorHandler from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,907 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002200: Adding class resource org.keycloak.services.resources.ThemeResource from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,907 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002205: Adding provider class org.keycloak.services.filters.KeycloakTransactionCommitter from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,907 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002200: Adding class resource org.keycloak.services.resources.JsResource from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,907 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.admin.AdminRoot from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,907 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.WelcomeResource from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,908 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.RobotsResource from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,908 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002210: Adding provider singleton org.keycloak.services.util.ObjectMapperResolver from Application class org.keycloak.services.resources.KeycloakApplication
09:34:09,908 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 68) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.RealmsResource from Application class org.keycloak.services.resources.KeycloakApplication
09:34:10,012 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 68) WFLYUT0021: Registered web context: '/auth' for server 'default-server'
09:34:10,133 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 43) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")
09:34:10,170 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
09:34:10,173 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
09:34:10,173 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
09:34:10,173 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.2 (WildFly Core 10.0.3.Final) started in 30805ms - Started 591 of 886 services (601 services are lazy, passive or on-demand)
09:34:28,065 INFO  [org.keycloak.events] (default task-1) type=LOGIN, realmId=master, clientId=admin-cli, userId=a3199bae-bb1c-4285-9aa9-2e05fba0b9a7, ipAddress=127.0.0.1, auth_method=openid-connect, token_id=157ea0b0-ae79-4603-8e49-113a5c2db169, grant_type=password, refresh_token_type=Refresh, scope='profile email', refresh_token_id=583e673a-6b76-4886-ac7d-993270260330, client_auth_method=client-secret, username=admin, authSessionParentId=39b6d7a9-a397-4eee-906d-79fa6d4b25ea, authSessionTabId=QM895Ubu37U
09:34:46,002 INFO  [org.keycloak.events] (default task-1) type=LOGIN, realmId=master, clientId=admin-cli, userId=a3199bae-bb1c-4285-9aa9-2e05fba0b9a7, ipAddress=127.0.0.1, auth_method=openid-connect, token_id=403d1265-c088-4dc2-a31c-08db923fa31d, grant_type=password, refresh_token_type=Refresh, scope='profile email', refresh_token_id=43b93f7b-7c57-4150-b279-50c3ec831fd7, client_auth_method=client-secret, username=admin, authSessionParentId=64b29193-1b7a-4eb7-852f-74df6b7887f8, authSessionTabId=31SKoBUU5W4
09:35:12,037 INFO  [org.keycloak.events] (default task-1) type=LOGIN, realmId=master, clientId=admin-cli, userId=a3199bae-bb1c-4285-9aa9-2e05fba0b9a7, ipAddress=127.0.0.1, auth_method=openid-connect, token_id=af37e8d6-28bb-497d-b8ab-c84694e3fb3c, grant_type=password, refresh_token_type=Refresh, scope='profile email', refresh_token_id=c012252e-8453-43f1-8e53-85ed2093aec0, client_auth_method=client-secret, username=admin, authSessionParentId=745ce0ba-07ac-46eb-872c-1bceda7a8a89, authSessionTabId=y501D5doVxI

Hi,
Thanks for replying with all the details.

Can you please do the following steps?
The same fix is planned for the next release.

  1. Edit the stateful set “pxcentral-keycloak” under namespace “px-backup” to have the following 2 changes.
    a. Under “spec/template/spec/containers/volumeMounts/”, add a new mountPath like
    - name: keycloakDir
    mountPath: /.keycloak
    b. Under “spec/template/spec/volumes/”, mount one empty directory like following
    - emptyDir: {}
    name: keycloakDir

For reference, you can check the diff at

  1. Rerun the post-install job again with the following command.

kubectl -n px-backup get job pxcentral-post-install-hook -o json | jq ‘del(.spec.selector)’ | jq ‘del(.spec.template.metadata.labels)’ | kubectl -n px-backup replace --force -f -

1 Like

It worked like a charm, thanks!
Looking forward the 1.2.2 stable release.
Cheers

1 Like