Today the closed-source giants maintain a substantial cost-per-token advantage due to proprietary inference tricks like speculative decoding kernels, ultra-optimized FlashAttention derivatives, multi-GPU pipeline orchestration and dynamic prefix-caching algorithms.
I'm not sure whether the DeepSeek's open-sourced inference engine does all of these, but it appears that plenty of inference tricks have just been revealed for the open source community to learn from and start improving upon.
I'm not sure whether the DeepSeek's open-sourced inference engine does all of these, but it appears that plenty of inference tricks have just been revealed for the open source community to learn from and start improving upon.