Broker token request not working from backend

So I have this Reactjs frontend, works all perfect, I can login using a provider. From the frontend I get a token from the keycloak token endpoint. Using this token I make a call to my own backend (Nestjs) and validate the token with the public key, till this point everything is working fine. Now my backend needs the access token from the broker and I know how to get it, I can even make the required request using postman. The request is as followed:

URL: http://keycloak.localhost/auth/realms/{realm}/broker/{provider}/token
Headers: “Authorization: Bearer {token}”

From postman this works and I get the token I need. When I do this from my backend it always returns 401. I have a feeling this is because of the origin, so because of that I added the backend origin to the keycloak client that the frontend uses. When I inspect the JWT token it even shows the origin of the backend in the “allowed-origins” array. Too bad it still doesn’t work, I checked the logs of keycloak and it showed the following:

The grey line of from postman and the orange one is from my backend. The only difference I can spot is the ip address, so I feel like I am in the right direction with the origin but I don’t know what to try anymore…

Hope this is enough information, any help appriciated!