I think (not 100% sure) Cillium [0][1] kinda already does this. This loophole is good for packet processing/routing and even introducing XDP based ACL to bypass any ip/nf tables and get that almost wire speed benefit. I use Cilium with these features for custom made k8s clusters with Talos OS without any kube-proxy.
[0]https://docs.cilium.io/en/stable/operations/performance/tuni...
[1]https://isovalent.com/blog/post/cilium-netkit-a-new-containe...