Getting token (DirectGrant) is very slow for 10K+ users

Hi all,
I read a lot about Keycloak performance issues with multiple realms and clients. therefore, I did not use multiple realms or clients but I need to have so many users, and that is the issue.
I only have 2 realms with 10 clients per each realm.

I use Keycloak 12.0.2 in Google Cloud with MySQL. my users are growing (now around 11.5K)
I experienced long response time to get a JWT using Direct Grant.
I used offline tokens to not store the sessions in the memory. what could be the problem?
I’m using default settings.
which solutions could be helpful for this matter? (to speedup token response)