We use Elixir at Tinfoil Security for our API Security Scanner, and it uses GenStage quite a bit under the hood. It's a big simplification, but we basically have a producer that emits a stream of "scan tasks" to be performed (things like "scan this endpoint for SQL injection"), and then those are consumed by individual worker processes.
It lets us super easily manage things like rate limiting and throttling, while providing backpressure so that an influx of scans (or scans on large APIs) won't overload our infrastructure.
It lets us super easily manage things like rate limiting and throttling, while providing backpressure so that an influx of scans (or scans on large APIs) won't overload our infrastructure.