Comment by Spivak

Buy why go through the extra effort compared to just having your servers ask the IdP for the user's keys?