Comment

Comment by iaw

vLLM supports "banning" certain tokens but I don't know if it can dynamically reduce them.

To my knowledge you can also "ban" with llama.cpp but it is passed in the API call rather than to the server at initialization.

parent iaw 4 months ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton