IMO the big win for Elixir/Nx/Bumblebee/etc is that you can do batched distributed inference out of the box without deploying anything separate to your app or hitting an API. Massive complexity reduction and you can more easily scale up or down. https://hexdocs.pm/nx/Nx.Serving.html#content
And there's also a scale to 0 story for when you're not using that GPU at all: https://github.com/phoenixframework/flame
1 language/toolchain. 1 deployable app. Real time and distributed machine learning baked in. 1 dev can go really far.