Nijikokun

Hey there, I work on the AI Gateway at ngrok, so take this with the appropriate grain of salt.

"LiteLLM plus a dashboard" is a reasonable thought. Especially since gateways generally share a lot of similar plumbing, and I think LiteLLM is a good open source option. If you only need a translation layer and are happy to operate one on your own I think that LiteLLM would be a good fit for you.

But, ngrok.ai is for teams that want to get started quickly without having to setup and scale anything themselves. I can tell you for certain it is not LiteLLM underneath the hood and relies on the same global networking infrastructure as ngrok.

Since it leverages the ngrok infrastructure, it can route to models running on your own infrastructure in addition to public models and comes baked in with per-key access controls, automatic failover, and usage visibility. How you want to scope your keys is up to you since they can represent people, applications, or environments.

We've definitely put thought into the primitives, and while it's pretty bare bones right now, we're continously improving it every day.

I'm always open to hearing about your needs, what you think we are missing, and how we could improve.

Feel free to reply here or message me at [email protected]

> only >> recenlty << started happening

can't believe I was baited into reading this slop

/jk

good post actually, and a fair point

I do think many people will argue that you can just not review things though.

Meta is behind a huge amount of it, they have funded the majority of these

thats why i use caddy or ngrok.ai

There was a lot of speculation whether or not the model would use them or even if it would attempt to use them and they noted this months ago. Now they have clear evidence of them doing so. Seems reasonable.

context windows are too small and conversely larger windows are not accurate enough its annoying

Defold is good, been around a very long time, but it's just rough around the edges, once you get used to it it's pretty smooth sailing though.

ngl building a social network isn't hard, getting people to use a social network is the hard part

it performs well with composition, however it seems SD and SDXL excels in capability and quality when intermixed with pipelines and workflows, this doesn't do much to talk about that comparison and whenever i see things like this i think about the overall workflow, like cool you do good composition but you don't fit within the workflow or ecosystem that surrounds that tool and thus i have low expectations around adoption

the biggest issue i've had for things like this is that ai doesn't understand context very well, anything that is beyond a context window creates hallucinations and it starts making up things that may exist in one location but it tries to apply it to a completely unrelated scenario, would be curious if this does understand the connected pieces appropriately and catches things that break those connections, otherwise it's just another linter?

I thought the main site was amazing, then I saw the docs. Absolutely amazing work. Well done. Extremely excited to try out WAAPI.

Type definitions are definitely compiled with CPP. Type erasure only happens with polymorphism types. You can actually view the types with ghidra.