How to make token expire when LOA expire time is up

Hello, I have configured ACR / LOA step up authentication as discussed in the official examples:

As I understand each LOA level has an expiry associated so that the level is only valid for x seconds.

My problem is that once I get the higher LOA level, when I use my refresh token the resulting access token still has the higher LOA level.

How do I make my token expire when the LOA level expires ?

Thank you for your help