Aperswal

Currently building a LLM router and got a few of my startup friends to try it out. But, I feel like their advice is going to be slightly biased because they don't wanna hurt my feelings. Not sure though if I should send out ads or emails or what to get beta testers.

It feels like if I post on reddit then all the subreddit rules would just be flagged and it wouldn't be allowed. And I don't wanna get my email thrown into some spam list.

How do you all get beta testers?

If you give an SLA of lets say 10 seconds to find cheaper inference. I look around different providers that offer flex/batch and try to fulfill in that time.

If I can't then I escalate to a default request.

This means you definitely get a response in 10 seconds (could still be a 5xx or 429 from default), but in those 10 seconds I might find you cheaper inference and drop costs by 50%, then charge 20% of that as FlexInference's cut, effectively giving you a 40% discount.

If I don't find the cheaper inference and have to escalate, then I don't charge at all, so it is free.

The basic logic for default routing or flex routing?

This isn't open sourced, as of now.

The routing is largely pass through or standard conversions between API surfaces.

The SDK uses the OpenAPI specs for it's given API surface.

The eval is in the hero of the website itself, everyday it updates with the current median cost saved and latency traded.

Additionally, there are a few demos I have added to the website right under the hero.

Hey everyone! My name is Adi, I am the founder of FlexInference.

I built this to reduce AI spend for all my side projects. I then turned it into a proxy for Anthropic, OpenAI, and Gemini. I also then built a python and typescript SDK with strict types and better error messages. This way I can do agentic coding with a bit less wrangling.

Now I am making it available to the public.

The way it works is you set a SLA in the "start_within" parameter. If it's default or auto then it gets sent as is to the provider. If you set a time between 5 seconds and 10 minutes then it first gets requested as a flex request. If that starts getting processed then great you get 50% off. If not then the router escalates it to a default request. This way you always get a response. But if possible in the SLA you provide, we will get you cheaper inference.

I made the router free because it just never made sense to me why the ones in the market charge 5.5%. It also supports Interactions, Messages, Chat Completions and Responses. Its deployed through Cloudflare workers so it's pretty fast. And you can see all your metrics from within the dashboard.

I would really love it if you all could try it out. It's free and it doesn't require a credit card until you need a "flex race". So if anything you can just save the fee you pay per request by using this instead for that.

Let me know if you have any suggestions, bugs, feature requests, or just wanna yell at me.

Thanks,

Adi