Keycloak only gets Google refresh token on first login

I’m using Keycloak with Google as identity provider. I need a refresh token from Google in order to manage the calendar of users. Here are my Keycloak Google IDP settings:

After login I fetch the refresh token according to https://www.keycloak.org/docs/latest/server_development/index.html#retrieving-external-idp-tokens. Which looks like this:

{
    "access_token": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "expires_in": 3599,
    "refresh_expires_in": 0,
    "refresh_token": "YYYYYYYYYYYYYYYYYYYYYYYYYYYY",
    "token_type": "Bearer",
    "id_token": "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZ",
    "not-before-policy": 0,
    "scope": "openid https://www.googleapis.com/auth/calendar https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile",
    "accessTokenExpiration": 1593706596
}

Now the problem is when I login a second time and then try to fetch the refresh token again it’s gone:

{
    "access_token": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "expires_in": 3599,
    "refresh_expires_in": 0,
    "token_type": "Bearer",
    "id_token": "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZ",
    "not-before-policy": 0,
    "scope": "https://www.googleapis.com/auth/calendar https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile openid",
    "accessTokenExpiration": 1593706782
}

I’m really not sure how this is possible. One thing that came to my mind is that Keycloak is not respecting the “Request refresh token” setting on subsequest logins, but I don’t know how to verify this. I also posted this on StackOverflow but got no answer: https://stackoverflow.com/questions/62700314/keycloak-only-gets-google-refresh-token-on-first-login.