I came here to post this. We make a lot of the same sorts of optimizations for our OS distro (debian based) -- disabling frequency scaling, core pinning, etc. Critically, CPU0 has a bunch of stuff you cannot push, and you're better off with using one of the other cores as an isolated island.
This is what the scheduler latency looks like on our isolated core:
# Total: 000300000
# Min Latencies: 00001
# Avg Latencies: 00005
# Max Latencies: 00059
# Histogram Overflows: 00000
This is what the scheduler latency looks like on our isolated core:
# Total: 000300000 # Min Latencies: 00001 # Avg Latencies: 00005 # Max Latencies: 00059 # Histogram Overflows: 00000
(those are uS!)