Use keycloak for only authentication and use custom filter for authorization (Spring boot)

I am trying to use keycloak just for authentication and have my own custom filter for authorization. So ideal flow will be: First Keycloak filter authenticates the request and sets authentication object in context. Then my custom filter should run and it should get that existing authentication object, add authorities in that authentication object and set it back in context.

My securityConfig is extending KeycloakWebSecurityConfigurerAdapter like this

@Configuration
@EnableWebSecurity
class SecurityConfig extends KeycloakWebSecurityConfigurerAdapter
{
    @Override
    protected void configure(HttpSecurity http) throws Exception
   {
      super.configure(http);
		http
		.cors()
		.and()
		.csrf().ignoringAntMatchers("/","/auth","/auth/logout").csrfTokenRepository(CookieCsrfTokenRepository.withHttpOnlyFalse()).and()
		.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS).and()
		.authorizeRequests()
		.antMatchers(
				"/",
				"/auth",
				"/password/**",
              "/register/**",
              "/v2/api-docs",
              "/actuator/**",
              "/configuration/ui",
              "/swagger-resources",
              "/configuration/security",
              "/swagger-ui.html",
              "/webjars/**",
              "/swagger-resources/configuration/ui",
              "/swagger-resources/configuration/security",
              "/browser/index.html#",
              "/browser/**").permitAll()
		.antMatchers(HttpMethod.POST, REGISTER).permitAll()
		.antMatchers(HttpMethod.POST, CONFIRM).permitAll()
		.anyRequest().authenticated()
		.and()
		.addFilter(new JWTAuthorizationFilter(authenticationManager(),context))
//		.addFilterAfter(new JWTAuthorizationFilter(authenticationManager(),context), KeycloakAuthenticationProcessingFilter.class)
		.headers()
	    .contentSecurityPolicy("script-src 'self'");
}

It runs KeycloakAuthenticationProcessingFilter first and then my custom filter(JWTAuthorizationFilter) but then it calls KeycloakAuthenticationProcessingFilter agains due to which authentication object is set again and authorities are cleared. (i tried couple of things. current code plus commented line and a few more)

So first of all is it the right way to use keycloak in speing boot application if so then how can i make it work that my filter runs last in the filter chain.