I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot.
* full kernel bypass (I even implemented driver for the networking hardware)
* everything that could disrupt the application disabled (like SME interrupts, etc.) Memory mapped as huge buffers to prevent tlb lookup failures, etc.
* application consists of threads pinned to specified cores
* each thread on the path of market data to sending the order is never calling the operating system for anything. When not processing anything it is busy spinning.
* all memory preallocated carefully to have it pinned to the local core
* data flows from the networking hardware into one core and then passess through cores using disruptor, each core doing further processing and publishing signals to the next core
* the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange.
* full kernel bypass (I even implemented driver for the networking hardware)
* everything that could disrupt the application disabled (like SME interrupts, etc.) Memory mapped as huge buffers to prevent tlb lookup failures, etc.
* application consists of threads pinned to specified cores
* each thread on the path of market data to sending the order is never calling the operating system for anything. When not processing anything it is busy spinning.
* all memory preallocated carefully to have it pinned to the local core
* data flows from the networking hardware into one core and then passess through cores using disruptor, each core doing further processing and publishing signals to the next core
* the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange.