We are getting a mysterious 403 unauthorized for one of our service account’s access token. Situation is basically this:
- Grab access token for service account from the POST /auth/realms/openid-connect/token endpoint. This service account has the following realm-management roles:
- manager-users
- query-groups
- query-users
- view-realm
- view-users
our access token lifetime is set to about 30 minutes for this realm, SSO session timeout is 4 hours and the SSO session idle is 12 hours. (long values, I know, but they’re experimental).
2. Get the first 50 users using the GET /auth/admin/realms//users?first=1&max=50 endpoint.
3. For each of these users, make 3 other calls to the admin API to get more information about them (groups, roles, etc).
This works well for the first 8 users, but for some reason when we’re making the 3rd per-user call on the the 9th user we are consistently getting a 403 forbidden error all of a sudden. Some additional facts:
- The same code works for 8 other users, so it is probably not the code.
- The 9th user is not significantly different from the other 8. He has largely the same properties and belongs to the same realm.
- The admin API documentation explicitly states (scroll to the next section “Rate Limiting,” the anchor to that section is broken so I had to link the section above it) that keycloak itself does no rate limiting, so that is not it. That documentation was last updated before our version of keycloak came out so it should be accurate.
- The access token or refresh tokens could not have timed out, as the call to get the 9th user happens seconds after the access token is generated and our token timeouts (given above) are much longer than that.
- Both servers on the same internal private subnet, and we are calling the keycloak server by its internal DNS name, so networking issues should not apply.
- Keycloak 9.0.3 talking to the tomcat 7 connector.
Please help me out of this jam, or at least give me some ideas, I am well and truly stuck!