Yeah. VLC for example doesn't have a frame-by-frame backward seek, because they argue it would blow up the memory consumption to keep all those frames around just in case you want to jump backward. And since videos are mostly hardware-decoded on the GPU these days, you'd be wasting all that memory in your VRAM.
Here's one of many threads where Jean-Baptiste Kempf addresses the feature request. https://forum.videolan.org/viewtopic.php?p=390778&sid=1a571d...
The amount of pausing is going to vary a lot by video. If you have an I-frame followed by 250 P-frames, there will probably be a noticeable pause.