You should not store any token in the frontend. Instead, have a server doing that and use a classic secure cookie session. The session is bound to the token / refresh token and can then do the token management.
Alternatively, don't use access token / refresh token but an ID token. The refresh of the token will be done via the ID token and the user's session (cookie) to the OAuth provider.
Alternatively, don't use access token / refresh token but an ID token. The refresh of the token will be done via the ID token and the user's session (cookie) to the OAuth provider.