Comment by harsh_neeraj

Just checked out the repo — looks good. Only thing I'd suggest is maybe plug in LiteLLM for a more generalized inference setup. Could make things more flexible