In lots of places this makes sense. E.g. lots of embedded ARM platforms have a separate AES / ECC accelerator peripheral.
The trouble comes when you need to share access to a memory mapped peripheral among multiple threads/processes/users etc. It can be done, but it's usually easier to manage CPU registers than peripheral devices for things like crypto operations in larger systems. Plus, you have to do access control to the peripheral (so other processes don't try and steal your key), if its all within the security boundary of a "normal" process, you get that (mostly) for free.
All of the above has caveats and exceptions, but generally (ARM, SPARC, x86, now RISC-V) take this approach.
The trouble comes when you need to share access to a memory mapped peripheral among multiple threads/processes/users etc. It can be done, but it's usually easier to manage CPU registers than peripheral devices for things like crypto operations in larger systems. Plus, you have to do access control to the peripheral (so other processes don't try and steal your key), if its all within the security boundary of a "normal" process, you get that (mostly) for free.
All of the above has caveats and exceptions, but generally (ARM, SPARC, x86, now RISC-V) take this approach.