Maybe you can only create rifts, and you need to create one before travelling back and forth. So only time travelling after the first time travel rift has been opened. Otherwise it'd break continuity.
EDIT: If you created such rift and nobody would come out, then you'd have to start worrying.
EDIT: If you created such rift and nobody would come out, then you'd have to start worrying.