If you want more convenience from Rust and do not want to mess with Rust borrow checker, you do not really have to switch to Swift: you can rely on Rust reference counting. Use 1.) Rust reference-counted smart pointers[1] for shareable immutable references, and 2.) Rust internal mutability[2] for non-shareable mutable references checked at runtime instead of compile time. Effectively, you will be writing kind of verbose Golang, but keep Rust expressiveness.
[1] https://doc.rust-lang.org/book/ch15-04-rc.html
[2] https://doc.rust-lang.org/book/ch15-05-interior-mutability.h...