Anecdotally, it's typically the people in the room who are either a) late to the meeting or b) go off on tangents or c) cause the meeting to overrun. There are lots of good and bad things about remote workers... but being a hamper on meetings shouldn't be one of them.
If you have 20 people in the office - and 1 remote worker - your conference times should be at the preferred time of the 20.
If you have 20 people in the office - and 1 remote worker - your conference times should be at the preferred time of the 20.