The pain of failures that are "hard to reason about once things went async" is real. Centralizing the retry/failover logic makes sense.
One pattern I've found useful: having a read-only view of what's actually hitting the wire before any retry logic kicks in. When you can see the raw request/response as it happens, you can tell whether the issue is your payload, the provider throttling, or something in between.
We built toran.sh for this - it's a transparent proxy that shows exactly what goes out and comes back in real-time. Different layer than what you're doing (you handle the orchestration, we just show the traffic), but they complement each other.
Curious how you handle visibility into what's actually being sent during partial stream failures?
One pattern I've found useful: having a read-only view of what's actually hitting the wire before any retry logic kicks in. When you can see the raw request/response as it happens, you can tell whether the issue is your payload, the provider throttling, or something in between.
We built toran.sh for this - it's a transparent proxy that shows exactly what goes out and comes back in real-time. Different layer than what you're doing (you handle the orchestration, we just show the traffic), but they complement each other.
Curious how you handle visibility into what's actually being sent during partial stream failures?