>>Booking.com go to some lengths to minimise the latency introduced by models, including horizontally scaled distributed copies of models, a in-house developed custom linear prediction engine, favouring models with fewer parameters, batching requests, and pre-computation and/or caching.
Any idea what these are ? especially the pre-computation/caching and batching. I'm not able to see what advantage does batching bring...or how you can really cache a prediction request
Any idea what these are ? especially the pre-computation/caching and batching. I'm not able to see what advantage does batching bring...or how you can really cache a prediction request