That policy covers in-game content too - the same page mentions:
"Steam does not contain any paid advertising, nor are advertising models supported in games distributed on Steam. There are varying interpretations for what constitutes ‘advertising’ in a game, so the examples below are meant to help guide developers as to what is and is not supported on Steam."
Under not supported:
"Developers should not utilize paid advertising as a business model in their game"
The Borderlands 4 adverts in BL3 are not paid adverts, and they don't advertise anything other than their own games.
"Steam does not contain any paid advertising, nor are advertising models supported in games distributed on Steam. There are varying interpretations for what constitutes ‘advertising’ in a game, so the examples below are meant to help guide developers as to what is and is not supported on Steam."
Under not supported: "Developers should not utilize paid advertising as a business model in their game"
The Borderlands 4 adverts in BL3 are not paid adverts, and they don't advertise anything other than their own games.