Comment by simplicio

Bilbo doesn't see him wear it, but he's described as having put it on to steal goblin children for food. So he does wear it in the books.

Its on such an expedition that the ring "slips" from him, further suggesting the ring is actually not only his size, but a little large.