The answer to your question is WebKit (because iOS), kernels (XNU, Linux, Windows) etc. In case you are not familiar with the domain I'd start with user-space exploitation and relevant write ups to get my feet wet. You'll find plenty of write ups, blogs etc. so I'll skip those.
Some of the books I generally found interesting are [1],[2], [3]. There's more to that, including fundamental concepts of CS (e.g., compilers and optimization in JITs, OS architecture etc.). I believe also https://p.ost2.fyi/dashboard has some relevant training.
[1] https://nostarch.com/zero-day
[2] https://nostarch.com/hacking2.htm
[3] https://ia801309.us.archive.org/26/items/Wiley.The.Shellcode...