I might be missing something here but assuming varnish is using mmap()+madvise(), accessing memory might block the thread until the page fault is served, which is not ideal for a user-facing server.
If you manage your own memory/swap, at least you can use async IO and free up the thread while the IO request is being served by the OS.
If you manage your own memory/swap, at least you can use async IO and free up the thread while the IO request is being served by the OS.