akarshc

I wrote this after building an AI gateway layer for ModelRiver to handle streaming responses, retries, and provider failover across LLM APIs. Phoenix supervision trees ended up being especially useful for isolating failures between model calls. Curious how others are structuring gateway layers in front of LLM providers.

I’ve worked on and shipped a few AI systems that reached real users.

This post isn’t about models or prompts. It’s about the things that kept breaking once AI moved off the happy path: async jobs, retries, silent failures, provider outages, cost blowups, and debugging without visibility.

I wrote this mostly as a way to document the mistakes I made and what I wish I had known earlier. Happy to answer questions or dig deeper into any of the failure modes.

This is one of the harder problems, and there isn’t a perfect answer.

The main thing we try to avoid is pretending mid-stream retries are the same as pre-request retries. Once a stream has started, we treat it as a sequence of events with checkpoints rather than a single opaque response. Retries are scoped to known safe boundaries, and anything ambiguous is surfaced explicitly instead of silently re-emitting tokens.

In other words, correctness is prioritized over pretending the stream is seamless. If we can’t guarantee no duplication, we make that visible rather than hide it.

This was something we were careful about. The request and event models are intentionally close to what most providers already expose, rather than introducing a completely new abstraction.

Teams usually integrate it incrementally in front of existing calls. If you remove it, you’re mostly deleting the orchestration layer and keeping your provider integrations and client logic. You lose centralized retries and observability, but you’re not stuck rewriting your entire request model.

If adopting it requires a full rewrite, that’s usually a sign it’s being applied too broadly.

I’m one of the builders. Once AI requests moved beyond simple sync calls, we kept running into the same problems in production: retries hiding failures, async flows that were hard to reason about, frontend state drifting, and providers timing out mid-request.

This page breaks down the three request patterns we see teams actually using in production (sync, async, and event-driven async), how data flows in each case, and why we ended up favoring an event-driven approach for interactive, streaming apps.

Happy to answer questions or go deeper on any part of the architecture.

Totally agree, that “what actually hit the wire?” view is critical once things go async.

ModelRiver already has this covered via request logs. Every request captures the full lifecycle, the exact payload sent to the provider, streaming chunks as they arrive, partial responses, errors, retries, and the final outcome. Even if a stream fails midway, you can still inspect what was sent and what came back before the failure.

So you can clearly tell whether the issue is payload shape, provider throttling, or a mid stream failure, before any retry or failover logic kicks in. That wire level visibility is core to how we approach debugging async AI requests.