Comment by IshKebab

The constraint is that the gears don't speed up and slow down (the gear ratio is constant). Also that the contact forces lie on a line.

There's a good animation on wikipedia:

https://en.wikipedia.org/wiki/Involute_gear

Involute gears are the only shape that satisfy those constraints.