It's because the organisms get trapped between your clothes and skin, and they only cause a problem in large numbers. Bare skin doesn't trap enough to do anything, usually. Also, when you rinse off, rinse off with saltwater. Fresh water kills them, which causes them to sting you.
Source: http://en.wikipedia.org/wiki/Seabather's_eruption