The syscall overhead isn't the problem, that's dirt cheap as you say. The problem is the context-switch overhead. Calling into the OS flushes a lot of data and instructions from the cache, and that lost performance after returning can easily add up to around 30µs.[1]
[1] http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-ma...