18.0.0 in EKS + ALB (StatusCode 401 after auth)

Hey all,

So I’m currently trying to configure Keycloak in AWS EKS + ALB (as my ingress) and I can’t wrap my head around the issue I’m having.

Infrastructure is as follows:
Browser HTTPS → ALB (all requests forwarded from HTTP to HTTPS) + OIDC configuration → Keycloak’ed application

Configuration is pretty basic and basically, all of it is from AWS/Keycloak documentation:
ALB annotations:

"alb.ingress.kubernetes.io/auth-type": "oidc",
"alb.ingress.kubernetes.io/auth-idp-oidc": (
    f'{{"issuer":"https://{self.keycloak_domain}/",'
    f'"authorizationEndpoint":"{auth_path}/protocol/openid-connect/auth",'
    f'"tokenEndpoint":"{auth_path}/protocol/openid-connect/token",'
    f'"userInfoEndpoint":"{auth_path}/protocol/openid-connect/userinfo",'
    f'"secretName":"{self.keycloak.secrets_name}"}}'
),
"alb.ingress.kubernetes.io/auth-on-unauthenticated-request": "authenticate",

Keycloak HELM values:

fullnameOverride: "{{ full_name }}"
extraStartupArgs: "--spi-login-protocol-openid-connect-legacy-logout-redirect-uri=true"

service:
  type: "ClusterIP"

auth:
  createAdminUser: true
  adminUser: "{{ admin_username }}"
  adminPassword: "{{ admin_password }}"

postgresql:
  enabled: false

externalDatabase:
  host: "{{ postgresql_host }}"
  database: "{{ postgresql_db }}"
  user: "{{ postgresql_user }}"
  password: "{{ postgresql_password }}"

extraEnvVars:
  - name: KEYCLOAK_HOSTNAME
    value: "{{ hostname }}"
  - name: PROXY_ADDRESS_FORWARDING
    value: "true"

ALB target groups point to HTTP endpoints for the applications.

When I try to enter keycloak’ed application in a new session I get the login screen and all, but right after I authenticate with a user I get a 401 Authorization Required message.

Some logs:

h2 2022-07-26T13:46:47.159916Z app/k8s-app-35696e2f02/c04ef09f288d7a34 IP:49848 10.0.2.19:8080 0.000 0.017 0.000 302 302 1802 3268 "GET https://keycloak.domain:443/realms/master/protocol/openid-connect/auth?client_id=client_id&redirect_uri=application.domain&response_type=code&scope=email&state=jVieMhHmn4yjZp0VP5sQ0dd0Rkd5MbV9sHHbN6x9DevQKpE5EWm7QuilY4YvvuIc%2FUMTN%2F427kc%2BAEFfrufUXZGQaL%2BRKHopIumEpnm6YIWPZFFGwuSE%2BPvWrAMnGkpU%2BL5lT3TUPmcH6angvPmEUfCFcSEl5cs2hTFp1A%2BlLRZUaJOCmBz9eBfm3AHNKgw3yTbPV2NTif4R98IVc9UL%2Bg%3D%3D HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:bla:targetgroup/k8s-keycloak "Root=1-62dff047-0f9f91cd130e7c2b7b49a233" "application.domain" "acm_cert_arn" 11 2022-07-26T13:46:47.142000Z "authenticate,forward" "-" "-" "10.0.2.19:8080" "302" "-" "-"
h2 2022-07-26T13:46:47.273614Z app/k8s-app-35696e2f02/c04ef09f288d7a34 IP:49848 - -1 -1 -1 401 - 341 616 "GET https://application.domain:443/oauth2/idpresponse?state=jVieMhHmn4yjZp0VP5sQ0dd0Rkd5MbV9sHHbN6x9DevQKpE5EWm7QuilY4YvvuIc%2FUMTN%2F427kc%2BAEFfrufUXZGQaL%2BRKHopIumEpnm6YIWPZFFGwuSE%2BPvWrAMnGkpU%2BL5lT3TUPmcH6angvPmEUfCFcSEl5cs2hTFp1A%2BlLRZUaJOCmBz9eBfm3AHNKgw3yTbPV2NTif4R98IVc9UL%2Bg%3D%3D&session_state=97a629e2-4cad-4b13-9509-70a921614d82&code=048522b5-d81a-4862-8319-73565322915a.97a629e2-4cad-4b13-9509-70a921614d82.398968b2-cc51-4f41-8680-92e32051d6b0 HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 - "Root=1-62dff047-3da776384a5c2ba110c6ac29" "application.domain" "acm_cert_arn" -1 2022-07-26T13:46:47.260000Z "authenticate" "-" "AuthInvalidTokenResponse" "-" "-" "-" "-"
2022-07-26 17:02:18,087 WARN  [org.keycloak.events] (executor-thread-37) type=CODE_TO_TOKEN_ERROR, realmId=realm_id, clientId=client_id, userId=null, ipAddress=ip_address, error=invalid_code, grant_type=authorization_code, code_id=7216dc67-fddc-4c27-ab5f-008f5a8c4c1c, client_auth_method=client-secret

What I’ve tried:

  1. ALB annotation “backend-protocol” HTTPS
  2. KEYCLOAK_FRONTEND_URL
  3. Playing with ALB target groups (HTTP vs HTTPS)

Anyone else has this setup? Did I miss something in configuring keycloak? ALB OIDC parses keycloak token response wrongfully?
Everything works fine when I run oauth2-proxy helm chart + nginx ingress.`

Thank you!