Yeah, one needs to understand that "unsafe" does not mark which parts of the code are actually unsafe, it simply marks parts where the compiler ignores parts of its rule set. But the implications can crop up anywhere, there is no guarantee that a resulting use-after-free or similar can only happen inside the unsafe blocks.
In short, if you use an unsafe block, then potentially any part of your code is unsafe.
In short, if you use an unsafe block, then potentially any part of your code is unsafe.