Yup, and besides, Go has the unsafe (https://golang.org/pkg/unsafe/) package, which allows for basically the same sort of unsafety as Rust's unsafe.
In fact, I would say Rust is typically more safe than Go, because in Rust you can mark any code as unsafe, it doesn't necessarily have to involve unsafe pointers. For example in Rust FFI functions are typically marked unsafe even if they don't involve pointers. There's no way to do that in Go.
Another example is strings: Rust ensures that strings are always valid UTF-8 by marking any function that could break that as unsafe. OTOH in Go you strings can contain invalid UTF-8 if I recall correctly.
In fact, I would say Rust is typically more safe than Go, because in Rust you can mark any code as unsafe, it doesn't necessarily have to involve unsafe pointers. For example in Rust FFI functions are typically marked unsafe even if they don't involve pointers. There's no way to do that in Go.
Another example is strings: Rust ensures that strings are always valid UTF-8 by marking any function that could break that as unsafe. OTOH in Go you strings can contain invalid UTF-8 if I recall correctly.