I don't see any provisions for scaling up workflows horizontally — is that something this supports, or this more for interactive agents?
As an example, I have a use case where I'm processing millions of documents in a fan-out/fan-in pipeline that uses multiple LLMs to analyze and condense information. This mostly isn't CPU-bound, but it does consume a lot of RAM.
I'm currently using Ray to split the workload across a cluster. Ray has autoscaling, so it's very good for this kind of thing.
As an example, I have a use case where I'm processing millions of documents in a fan-out/fan-in pipeline that uses multiple LLMs to analyze and condense information. This mostly isn't CPU-bound, but it does consume a lot of RAM.
I'm currently using Ray to split the workload across a cluster. Ray has autoscaling, so it's very good for this kind of thing.