Next time tell him that if he isn't going to provide something constructive to the meeting, that he won't be allowed to join further ones.
As long as you make it clear from your tone and wording, that this is not you being aggressive in any way but you protecting the time of the team from distractions, the rest of the team shouldn't see you any different than before.
As long as you make it clear from your tone and wording, that this is not you being aggressive in any way but you protecting the time of the team from distractions, the rest of the team shouldn't see you any different than before.