403 using FeignClient, RequestInterceptor and Bearer Token

Hello Keycloak Community,

I am trying to integrate keycloak with several spring boot microservices + spring security.
I watched Thomas Darimont’s excellent spring/keycloak presentation on YouTube and downloaded his github code as a guide…The initial login and authorization works fine, but I am not able to get the service to service authorization working…here are the details:

I am using java 11 & kubernetes running on minikube. I deployed keycloak using the latest Helm chart from codecentric/keycloak. I’m using the following dependencies:

spring-boot-starter-security
keycloak-spring-boot-starter
keycloak-spring-security-adapter

versioned at spring cloud Hoxton.SR6 and keycloak-adapter-bom 11.0.0.

I can access the first microservice ok, getting redirected to keycloak, logging in and then getting redirected back to the first microservice when the user has a role with permissisons for the URL. This microservice is set up as a public client.
So far so good.

Then I attempt to access a second microservice from the first microservice using spring cloud openfeign.
I created a feign.RequestInterceptor as detailed here: https://github.com/thomasdarimont/keycloak-docker-demo/blob/master/keycloak-demos/spring-boot-frontend/src/main/java/demo/todo/TodoClientConfig.java.

In the backend service with the feign implementations, I set up keycloak in the yaml as follows:

keycloak:
realm: myRealm
auth-server-url: url

resource: mymicroservice
credentials:
secret: …
bearer-only: true
ssl-required: external
principal-attribute: subject
use-resource-role-mappings: true

The backend service is also configured with config file that extends KeycloakWebSecurityConfigurerAdapter, almost identical to the configuration that the first microservice has, except that it uses a NullAuthenticatedSessionStrategy.

When I configure the backed to just authenticate all users the services work and return data.
However, once I configure the endpoints with role permissions like the the following (same roles as the first microservice had to authorize its URLs):

@Override
protected void configure(HttpSecurity http) throws Exception
{
    super.configure(http);
    http.authorizeRequests()
        .antMatchers("/**").hasAnyRole("Developers")
        .anyRequest()
        .permitAll();
}

I see the following feign exceptions:

2020-09-22 20:24:00.967 ERROR 1 — [nio-8080-exec-3] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is feign.FeignException$Forbidden: [403] during [GET] to [http://greeting-service/en-US] [GreetingService#getGreeting(String)]: [{“timestamp”:“2020-09-22T20:24:00.896+00:00”,“status”:403,“error”:“Forbidden”,“message”:"",“path”:"/en-US"}]] with root cause

feign.FeignException$Forbidden: [403] during [GET] to [http://greeting-service/en-US] [GreetingService#getGreeting(String)]: [{“timestamp”:“2020-09-22T20:24:00.896+00:00”,“status”:403,“error”:“Forbidden”,“message”:"",“path”:"/en-US"}]

2 other things:

I also see the following warning multiple times in the logs:
2020-09-22 20:24:00.991 WARN 1 — [nio-8080-exec-3] o.k.c.u.DelegatingSerializationFilter : Could not set ObjectFilter: null

The full stacktrace:

2020-09-22 20:24:00.967 ERROR 1 — [nio-8080-exec-3] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path threw exception [Request processing failed; nested exception is feign.FeignException$Forbidden: [403] during [GET] to [http://greeting-service/en-US] [GreetingService#getGreeting(String)]: [{“timestamp”:“2020-09-22T20:24:00.896+00:00”,“status”:403,“error”:“Forbidden”,“message”:“”,“path”:“/en-US”}]] with root cause
feign.FeignException$Forbidden: [403] during [GET] to [http://greeting-service/en-US] [GreetingService#getGreeting(String)]: [{“timestamp”:“2020-09-22T20:24:00.896+00:00”,“status”:403,“error”:“Forbidden”,“message”:“”,“path”:“/en-US”}]
at feign.FeignException.clientErrorStatus(FeignException.java:199) ~[feign-core-10.10.1.jar:na]
at feign.FeignException.errorStatus(FeignException.java:177) ~[feign-core-10.10.1.jar:na]
at feign.FeignException.errorStatus(FeignException.java:169) ~[feign-core-10.10.1.jar:na]
at feign.codec.ErrorDecoder$Default.decode(ErrorDecoder.java:92) ~[feign-core-10.10.1.jar:na]
at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96) ~[feign-core-10.10.1.jar:na]
at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138) ~[feign-core-10.10.1.jar:na]
at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89) ~[feign-core-10.10.1.jar:na]
at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100) ~[feign-core-10.10.1.jar:na]
at com.sun.proxy.$Proxy123.getGreeting(Unknown Source) ~[na:na]
at com.acmecompany.dbg.demo.web.controller.WebController._buildGreeting(WebController.java:84) ~[classes/:na]
at com.acmecompany.dbg.demo.web.controller.WebController.home(WebController.java:69) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) ~[tomcat-embed-core-9.0.37.jar:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[tomcat-embed-core-9.0.37.jar:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticatedActionsFilter.doFilter(KeycloakAuthenticatedActionsFilter.java:57) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.keycloak.adapters.springsecurity.filter.KeycloakSecurityContextRequestFilter.doFilter(KeycloakSecurityContextRequestFilter.java:61) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.keycloak.adapters.springsecurity.filter.KeycloakPreAuthActionsFilter.doFilter(KeycloakPreAuthActionsFilter.java:96) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticatedActionsFilter.doFilter(KeycloakAuthenticatedActionsFilter.java:74) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.keycloak.adapters.springsecurity.filter.KeycloakSecurityContextRequestFilter.doFilter(KeycloakSecurityContextRequestFilter.java:92) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.keycloak.adapters.springsecurity.filter.KeycloakPreAuthActionsFilter.doFilter(KeycloakPreAuthActionsFilter.java:96) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) ~[spring-session-core-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) ~[spring-session-core-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) ~[spring-boot-actuator-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.keycloak.adapters.tomcat.AbstractAuthenticatedActionsValve.invoke(AbstractAuthenticatedActionsValve.java:67) ~[spring-boot-container-bundle-11.0.0.jar:11.0.0]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.invoke(AbstractKeycloakAuthenticatorValve.java:181) ~[spring-boot-container-bundle-11.0.0.jar:11.0.0]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

I am including some logs from keycloak when the web app invokes the backend app and gets a 403.
Also, an access token that is generated through Postman…
I have not really customized roles or clients - just created a confidential web client and bearer only backend client. A couple of users and roles (Developers is used in this case and I am expecting to pass authorization…)
Are JWT Decoders required on the backend serive in order to correctly parse the JWT…?

16:01:42,669 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-16) processFlow: browser
16:01:42,674 DEBUG [org.keycloak.services.managers.AuthenticationSessionManager] (default task-16) Removing authSession ‘6e2a1eb2-2f08-4015-9f7d-8bac42c9d3a9’. Expire restart cookie: true
16:01:42,675 DEBUG [org.keycloak.events] (default task-16) type=LOGIN, realmId=webRealm, clientId=web, userId=6b5e41e1-3267-4e88-ac0d-ff3c1934a848, ipAddress=XXX.XX.X.X, auth_method=openid-connect, auth_type=code, redirect_uri=http://XXX.XXX.XX.XX:30267/sso/login, consent=no_consent_required, code_id=6e2a1eb2-2f08-4015-9f7d-8bac42c9d3a9, username=smurphy, authSessionParentId=6e2a1eb2-2f08-4015-9f7d-8bac42c9d3a9, authSessionTabId=Hi3aVaWaG7w
16:01:42,675 DEBUG [org.keycloak.services.util.CookieHelper] (default task-16) Couldnt find cookie {0}, trying {1}
16:01:42,675 DEBUG [org.keycloak.services.managers.AuthenticationManager] (default task-16) Create login cookie - name: KEYCLOAK_IDENTITY, path: /auth/realms/webRealm/, max-age: -1
16:01:42,676 DEBUG [org.keycloak.services.managers.AuthenticationManager] (default task-16) Expiring remember me cookie
16:01:42,676 DEBUG [org.keycloak.services.managers.AuthenticationManager] (default task-16) Expiring cookie: KEYCLOAK_REMEMBER_ME path: /auth/realms/webRealm/
16:01:42,676 DEBUG [org.keycloak.protocol.oidc.OIDCLoginProtocol] (default task-16) redirectAccessCode: state: fe3c4abc-92e9-45ac-bdfb-4b26e7f2cda2
16:01:42,677 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-16) JtaTransactionWrapper commit
16:01:42,678 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-16) JtaTransactionWrapper end
16:01:42,715 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-16) new JtaTransactionWrapper
16:01:42,715 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-16) was existing? false
16:01:42,716 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (default task-16) AUTHENTICATE CLIENT
16:01:42,716 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-16) client authenticator: client-secret
16:01:42,716 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-16) client authenticator SUCCESS: client-secret
16:01:42,716 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-16) Client web authenticated by client-secret
16:01:42,717 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider] (default task-16) getUserSessionWithPredicate(6e2a1eb2-2f08-4015-9f7d-8bac42c9d3a9): found in local cache
16:01:42,717 DEBUG [org.keycloak.protocol.oidc.endpoints.TokenEndpoint] (default task-16) Adapter Session ‘66559347-b0b9-4141-a9c7-760626b717c7’ saved in ClientSession for client ‘web’. Host is ‘web-6864bf799f-m9mb4’
16:01:42,736 DEBUG [org.keycloak.events] (default task-16) type=CODE_TO_TOKEN, realmId=webRealm, clientId=web, userId=6b5e41e1-3267-4e88-ac0d-ff3c1934a848, ipAddress=172.17.0.8, token_id=838dccdf-017f-44e1-826f-7127b69e1dd2, grant_type=authorization_code, refresh_token_type=Refresh, scope=‘openid profile email’, refresh_token_id=63aa8345-bd12-40f8-a030-77bd8234bfb3, code_id=6e2a1eb2-2f08-4015-9f7d-8bac42c9d3a9, client_auth_method=client-secret

{
“exp”: 1601312021,
“iat”: 1601311721,
“auth_time”: 1601311721,
“jti”: “43f716f5-a641-4bd2-ae6b-d010fa5bfc43”,
“iss”: “http://keycloak.XXX.XXX.XX.XX.nip.io/auth/realms/webRealm”,
“aud”: “account”,
“sub”: “6b5e41e1-3267-4e88-ac0d-ff3c1934a848”,
“typ”: “Bearer”,
“azp”: “minky”,
“session_state”: “f9687a80-294d-4904-84e2-38270426cd4b”,
“acr”: “1”,
“realm_access”: {
“roles”: [
“Developers”,
“offline_access”,
“uma_authorization”
]
},
“resource_access”: {
“account”: {
“roles”: [
“manage-account”,
“manage-account-links”,
“view-profile”
]
}
},
“scope”: “openid profile email”,
“email_verified”: false,
“preferred_username”: “smurphy”
}

Here are some additional logs that I am now getting…still trying to understand what they mean exactly :

2020-09-29 19:34:50.174 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 1 of 15 in additional filter chain; firing Filter: ‘WebAsyncManagerIntegrationFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 2 of 15 in additional filter chain; firing Filter: ’ ’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] w.c.HttpSessionSecurityContextRepository : Obtained a valid SecurityContext from SPRING_SECURITY_CONTEXT: ‘org.springframework.security.core.context.SecurityContextImpl@ea1cdc66: Authentication: org.keycloak.adapters.springsecurity.token.KeycloakAuthenticationToken@ea1cdc66: Principal: smurphy; Credentials: [PROTECTED]; Authenticated: true; Details: org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@362c6c9b; Granted Authorities: ROLE_offline_access, ROLE_Developers, ROLE_uma_authorization’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 3 of 15 in additional filter chain; firing Filter: ‘HeaderWriterFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 4 of 15 in additional filter chain; firing Filter: ‘CsrfFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 5 of 15 in additional filter chain; firing Filter: ‘KeycloakPreAuthActionsFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.k.adapters.PreAuthActionsHandler : adminRequest http://192.168.64.40:31294/error
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 6 of 15 in additional filter chain; firing Filter: ‘KeycloakAuthenticationProcessingFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using Ant [pattern=‘/sso/login’]
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Checking match of request : ‘/error’; against ‘/sso/login’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using RequestHeaderRequestMatcher [expectedHeaderName=Authorization, expectedHeaderValue=null]
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using org.keycloak.adapters.springsecurity.filter.QueryParamPresenceRequestMatcher@65b13bef
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using org.keycloak.adapters.springsecurity.filter.AdapterStateCookieRequestMatcher@68bb5070
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : No matches found
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 7 of 15 in additional filter chain; firing Filter: ‘LogoutFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Request ‘GET /error’ doesn’t match ‘POST /sso/logout’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 8 of 15 in additional filter chain; firing Filter: ‘RequestCacheAwareFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.s.HttpSessionRequestCache : saved request doesn’t match
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 9 of 15 in additional filter chain; firing Filter: ‘SecurityContextHolderAwareRequestFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 10 of 15 in additional filter chain; firing Filter: ‘KeycloakSecurityContextRequestFilter’
2020-09-29 19:34:50.175 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 11 of 15 in additional filter chain; firing Filter: ‘KeycloakAuthenticatedActionsFilter’
2020-09-29 19:34:50.176 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 12 of 15 in additional filter chain; firing Filter: ‘AnonymousAuthenticationFilter’
2020-09-29 19:34:50.176 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.a.AnonymousAuthenticationFilter : SecurityContextHolder not populated with anonymous token, as it already contained: ‘org.keycloak.adapters.springsecurity.token.KeycloakAuthenticationToken@ea1cdc66: Principal: smurphy; Credentials: [PROTECTED]; Authenticated: true; Details: org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@362c6c9b; Granted Authorities: ROLE_offline_access, ROLE_Developers, ROLE_uma_authorization’
2020-09-29 19:34:50.176 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 13 of 15 in additional filter chain; firing Filter: ‘SessionManagementFilter’
2020-09-29 19:34:50.176 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 14 of 15 in additional filter chain; firing Filter: ‘ExceptionTranslationFilter’
2020-09-29 19:34:50.176 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error at position 15 of 15 in additional filter chain; firing Filter: ‘FilterSecurityInterceptor’
2020-09-29 19:34:50.176 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /error reached end of additional filter chain; proceeding with original chain
2020-09-29 19:34:50.176 DEBUG 1 — [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : “ERROR” dispatch for GET “/error”, parameters={}
2020-09-29 19:34:50.177 DEBUG 1 — [nio-8080-exec-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletResponse)
2020-09-29 19:34:50.178 DEBUG 1 — [nio-8080-exec-1] o.s.w.s.v.ContentNegotiatingViewResolver : Selected ‘text/html’ given [text/html, text/html;q=0.8]
2020-09-29 19:34:50.179 DEBUG 1 — [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Exiting from “ERROR” dispatch, status 500
2020-09-29 19:34:50.179 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.a.ExceptionTranslationFilter : Chain processed normally
2020-09-29 19:34:50.179 DEBUG 1 — [nio-8080-exec-1] s.s.w.c.SecurityContextPersistenceFilter : SecurityContextHolder now cleared, as request processing completed

Attached are more logs and the AccessDeniedException.
My understanding of these logs is as follows…:

The token is verified and the user is authenticated.
The ID for the user does correspond to the logged in user’s ID in the Keycloak Admin Console.
The granted authorities of ROLE_UMA_PROTECTION in the KeycloakAuthenticationToken is not what I expect to see. I expect to see the ROLE_Developer that is associated with this user in the Admin Console.
Spring security is configured to check the URI “en-US” against ROLE_Developer, but it looks like that is not a role in the token. That role is in the token that is put into the authorization header using the Feign RequestInterceptor, so it seems like the backend as configured is not reading the roles from the token and creating granted authorities from them.
I am not using any spring oauth2 dependencies, just spring-boot-starter-security - do I need to take extra steps to parse/decode the token more thoroughly into something that spring can use correctly?

2020-10-01 20:14:30.537 DEBUG 1 — [nio-8080-exec-1] o.k.adapters.PreAuthActionsHandler : adminRequest http://172.17.0.7:8080/en-US
2020-10-01 20:14:30.537 DEBUG 1 — [nio-8080-exec-1] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /en-US
2020-10-01 20:14:30.537 DEBUG 1 — [nio-8080-exec-1] o.k.a.AuthenticatedActionsHandler : AuthenticatedActionsValve.invoke http://172.17.0.7:8080/en-US
2020-10-01 20:14:30.537 DEBUG 1 — [nio-8080-exec-1] o.k.a.AuthenticatedActionsHandler : Policy enforcement is disabled.
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 1 of 15 in additional filter chain; firing Filter: ‘WebAsyncManagerIntegrationFilter’
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 2 of 15 in additional filter chain; firing Filter: ‘SecurityContextPersistenceFilter’
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] w.c.HttpSessionSecurityContextRepository : No HttpSession currently exists
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] w.c.HttpSessionSecurityContextRepository : No SecurityContext was available from the HttpSession: null. A new one will be created.
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 3 of 15 in additional filter chain; firing Filter: ‘HeaderWriterFilter’
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 4 of 15 in additional filter chain; firing Filter: ‘CsrfFilter’
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 5 of 15 in additional filter chain; firing Filter: ‘KeycloakPreAuthActionsFilter’
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.k.adapters.PreAuthActionsHandler : adminRequest http://172.17.0.7:8080/en-US
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 6 of 15 in additional filter chain; firing Filter: ‘KeycloakAuthenticationProcessingFilter’
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using Ant [pattern=‘/sso/login’]
2020-10-01 20:14:30.538 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Checking match of request : ‘/en-US’; against ‘/sso/login’
2020-10-01 20:14:30.539 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using RequestHeaderRequestMatcher [expectedHeaderName=Authorization, expectedHeaderValue=null]
2020-10-01 20:14:30.539 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : matched
2020-10-01 20:14:30.539 DEBUG 1 — [nio-8080-exec-1] f.KeycloakAuthenticationProcessingFilter : Request is to process authentication
2020-10-01 20:14:30.539 DEBUG 1 — [nio-8080-exec-1] f.KeycloakAuthenticationProcessingFilter : Attempting Keycloak authentication
2020-10-01 20:14:30.539 DEBUG 1 — [nio-8080-exec-1] o.k.a.BearerTokenRequestAuthenticator : Found [1] values in authorization header, selecting the first value for Bearer.
2020-10-01 20:14:30.539 DEBUG 1 — [nio-8080-exec-1] o.k.a.BearerTokenRequestAuthenticator : Verifying access_token
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.k.a.BearerTokenRequestAuthenticator : successful authorized
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] a.s.a.SpringSecurityRequestAuthenticator : Completing bearer authentication. Bearer roles: [uma_protection]
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.k.adapters.RequestAuthenticator : User ‘6b5e41e1-3267-4e88-ac0d-ff3c1934a848’ invoking ‘http://172.17.0.7:8080/en-US’ on client ‘greeting’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.k.adapters.RequestAuthenticator : Bearer AUTHENTICATED
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] f.KeycloakAuthenticationProcessingFilter : Auth outcome: AUTHENTICATED
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.s.authentication.ProviderManager : Authentication attempt using org.keycloak.adapters.springsecurity.authentication.KeycloakAuthenticationProvider
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] f.KeycloakAuthenticationProcessingFilter : Authentication success using bearer token/basic authentication. Updating SecurityContextHolder to contain: org.keycloak.adapters.springsecurity.token.KeycloakAuthenticationToken@5a237d65: Principal: 6b5e41e1-3267-4e88-ac0d-ff3c1934a848; Credentials: [PROTECTED]; Authenticated: true; Details: org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@5176312; Granted Authorities: ROLE_UMA_PROTECTION
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 7 of 15 in additional filter chain; firing Filter: ‘LogoutFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Request ‘GET /en-US’ doesn’t match ‘POST /sso/logout’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 8 of 15 in additional filter chain; firing Filter: ‘RequestCacheAwareFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.s.HttpSessionRequestCache : saved request doesn’t match
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 9 of 15 in additional filter chain; firing Filter: ‘SecurityContextHolderAwareRequestFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 10 of 15 in additional filter chain; firing Filter: ‘KeycloakSecurityContextRequestFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 11 of 15 in additional filter chain; firing Filter: ‘KeycloakAuthenticatedActionsFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.k.a.AuthenticatedActionsHandler : AuthenticatedActionsValve.invoke http://172.17.0.7:8080/en-US
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.k.a.AuthenticatedActionsHandler : Policy enforcement is disabled.
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 12 of 15 in additional filter chain; firing Filter: ‘AnonymousAuthenticationFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.a.AnonymousAuthenticationFilter : SecurityContextHolder not populated with anonymous token, as it already contained: ‘org.keycloak.adapters.springsecurity.token.KeycloakAuthenticationToken@5a237d65: Principal: 6b5e41e1-3267-4e88-ac0d-ff3c1934a848; Credentials: [PROTECTED]; Authenticated: true; Details: org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@5176312; Granted Authorities: ROLE_UMA_PROTECTION’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 13 of 15 in additional filter chain; firing Filter: ‘SessionManagementFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] s.CompositeSessionAuthenticationStrategy : Delegating to org.springframework.security.web.authentication.session.NullAuthenticatedSessionStrategy@163fbbc9
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] s.CompositeSessionAuthenticationStrategy : Delegating to org.springframework.security.web.csrf.CsrfAuthenticationStrategy@c280a2d
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] w.c.HttpSessionSecurityContextRepository : HttpSession being created as SecurityContext is non-default
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] w.c.HttpSessionSecurityContextRepository : SecurityContext ‘org.springframework.security.core.context.SecurityContextImpl@5a237d65: Authentication: org.keycloak.adapters.springsecurity.token.KeycloakAuthenticationToken@5a237d65: Principal: 6b5e41e1-3267-4e88-ac0d-ff3c1934a848; Credentials: [PROTECTED]; Authenticated: true; Details: org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@5176312; Granted Authorities: ROLE_UMA_PROTECTION’ stored to HttpSession: ‘org.apache.catalina.session.StandardSessionFacade@1c1b2d94
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 14 of 15 in additional filter chain; firing Filter: ‘ExceptionTranslationFilter’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.security.web.FilterChainProxy : /en-US at position 15 of 15 in additional filter chain; firing Filter: ‘FilterSecurityInterceptor’
2020-10-01 20:14:30.557 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Request ‘GET /en-US’ doesn’t match ‘POST /sso/logout’
2020-10-01 20:14:30.558 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using Ant [pattern=’/actuator/health/']
2020-10-01 20:14:30.558 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Checking match of request : ‘/en-US’; against '/actuator/health/

2020-10-01 20:14:30.558 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using Ant [pattern=‘/actuator/info/']
2020-10-01 20:14:30.558 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Checking match of request : ‘/en-US’; against '/actuator/info/

2020-10-01 20:14:30.558 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using Ant [pattern=‘/actuator’]
2020-10-01 20:14:30.558 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Checking match of request : ‘/en-US’; against ‘/actuator’
2020-10-01 20:14:30.561 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : Trying to match using Ant [pattern=‘/actuator/’]
2020-10-01 20:14:30.562 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Checking match of request : ‘/en-US’; against ‘/actuator/’
2020-10-01 20:14:30.562 DEBUG 1 — [nio-8080-exec-1] o.s.s.web.util.matcher.OrRequestMatcher : No matches found
2020-10-01 20:14:30.562 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.u.matcher.AntPathRequestMatcher : Checking match of request : ‘/en-US’; against ‘/en-US’
2020-10-01 20:14:30.563 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.a.i.FilterSecurityInterceptor : Secure object: FilterInvocation: URL: /en-US; Attributes: [hasRole(‘ROLE_Developers’)]
2020-10-01 20:14:30.563 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.a.i.FilterSecurityInterceptor : Previously Authenticated: org.keycloak.adapters.springsecurity.token.KeycloakAuthenticationToken@5a237d65: Principal: 6b5e41e1-3267-4e88-ac0d-ff3c1934a848; Credentials: [PROTECTED]; Authenticated: true; Details: org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@5176312; Granted Authorities: ROLE_UMA_PROTECTION
2020-10-01 20:14:30.563 DEBUG 1 — [nio-8080-exec-1] o.s.s.access.vote.AffirmativeBased : Voter: org.springframework.security.web.access.expression.WebExpressionVoter@6e4c9c65, returned: -1
2020-10-01 20:14:30.567 DEBUG 1 — [nio-8080-exec-1] o.s.s.w.a.ExceptionTranslationFilter : Access is denied (user is not anonymous); delegating to AccessDeniedHandler
org.springframework.security.access.AccessDeniedException: Access is denied
at org.springframework.security.access.vote.AffirmativeBased.decide(AffirmativeBased.java:84) ~[spring-security-core-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.access.intercept.AbstractSecurityInterceptor.beforeInvocation(AbstractSecurityInterceptor.java:233) ~[spring-security-core-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:123) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticatedActionsFilter.doFilter(KeycloakAuthenticatedActionsFilter.java:74) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.keycloak.adapters.springsecurity.filter.KeycloakSecurityContextRequestFilter.doFilter(KeycloakSecurityContextRequestFilter.java:92) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticationProcessingFilter.successfulAuthentication(KeycloakAuthenticationProcessingFilter.java:214) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:240) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.keycloak.adapters.springsecurity.filter.KeycloakPreAuthActionsFilter.doFilter(KeycloakPreAuthActionsFilter.java:96) ~[keycloak-spring-security-adapter-11.0.0.jar:11.0.0]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.3.3.RELEASE.jar:5.3.3.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) ~[spring-boot-actuator-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.keycloak.adapters.tomcat.AbstractAuthenticatedActionsValve.invoke(AbstractAuthenticatedActionsValve.java:67) ~[spring-boot-container-bundle-11.0.0.jar:11.0.0]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.invoke(AbstractKeycloakAuthenticatorValve.java:181) ~[spring-boot-container-bundle-11.0.0.jar:11.0.0]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

I finally got this working.

I watched a bunch of youtubes & medium.com articles and put together a rudimentary straw man.
Long story short, from all that chaos, I ended up having the following property:
use-resource-role-mappings: true
It needed to be set to false, otherwise the AdapterUtils in SpringSecurityRequestAuthenticator will not read the realm roles, which are all I had set from battling through the indoctrinatory material. I am only nebulously aware of resource roles, at this stage of my evolution.

Also, there are snippets out there on the internet-thing that, in configuration, set the SimpleAuthorityMapper property:
simpleAuthorityMapper.setConvertToUpperCase(true);

I found that this was not right - spring 5.3.3 WebExpressionVoter used “ROLE_lowercase” to compare roles…