This is such a great post. It really shows how much room for improvement there is in all released deep learning code. Almost none of the open source work is really production ready for fast inference, and tuning the systems requires a good working knowledge of the GPU.
The article does skip the most important step for getting great inference speeds: Drop Python and move fully into C++.
The article does skip the most important step for getting great inference speeds: Drop Python and move fully into C++.