Comment by akshay_akula

You guys should look into ngrok ai gateway. We have some small models running on local hardware that we tried to use but it was too painful. Just wanted to not waste all the compute hit one endpoint and be done but as a team.