If you do that, then on almost every context switch the new process will resume with a cold cache. In the worst case, if you were to send lots of IPC requests to a privileged process then your process could have a cold cache far too often.
Many older generations of ARM (v5 and earlier) used a virtually indexed, virtually tagged (VIVT) cache to reduce latency but this required an L1 flush on every context switch and that was incredibly painful!
Many older generations of ARM (v5 and earlier) used a virtually indexed, virtually tagged (VIVT) cache to reduce latency but this required an L1 flush on every context switch and that was incredibly painful!