aarnphm

Hi srameshc, the core of BentoML is still considered MLOps. A lot of our customers are pretty much MLOps users. However, LLMOps seem like a natural progression of the product, given that a lot of our users now want to experiment/build with LLM-based services.

Thanks for the recommendation, I'm actually working on something similar for this part of the docs (I'm also working at BentoML).

You can pretty much make the same argument about Docker. Docker abstracts away runc, runc abstract away cgroup.

I don't think calling it abstraction is correct. OneDiffusion is designed to be opinionated and help users to run diffusion models easily. It includes best practice, default options, optimisation baked in such that it helps developers to move faster.

You can also make that argument about debugging any AI application in production. If you were just writing a simple web server to run this model behind a APIs, there are a plethora to think about, scaling, resource utilisation, load balancing, batching, etc. Hence the argument for creating an standarized interface for these type of serving problems.

We haven't even touched k8s, and throughput, latency, serving optimisation, but you get the idea of building this from scratch versus using library such as OneDiffusion.

Of course we are also working to improve the library and include more features, so looking forward to hearing more constructive feedback!

Currently on main, 8bit and 4bit quant is supported

One can simply do

```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```

Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization

Hi there, 8bit and 4bit is currently supported on main. GPTQ is working in progress, as well as GGML

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

Again?? I definitely have to try out BentoML then to see what the hype is about!!

BentoML is an amazing tool that allows you to quickly and easily deploy your machine learning models as APIs. It is extremely user-friendly and easy to use, and the results are amazing. I highly recommend BentoML for anyone who wants to deploy their machine learning models quickly and easily.