Comment by mihaich

I made a simple framework for LLM sampling algorithms that can discard generated tokens.

This means it gives you the ability to set rules by which the last tokens are considered incorrect and need to be regenerated.

I have included 2 demo algorithms.

It offers support for both GGUF models (llama.cpp) and models in Huggingface format (Transformers library).

Enjoy!