mrciffa

It should work with any type of model, obviously longer chain of thoughts will be more difficult to analyse by the evaluation model, because it will have way more reasoning steps to identify and separate. The quality of the outcome depends a lot on the chosen model to give you insights. We tested with Llama3-70B and worked smoothly most of the times.

We are currently giving broad suggestions with an insight model that can be chosen during the setup. We will try to update and improve the suggestion prompt/code to make them more granular with new releases

Yes, reasoning models can potentially be optimized with our uncertainty estimations. We are currently testing the library with DeepSeek R1

Unfortunately LLMs are a gigantic monster to understand, we were considering your same approach with sliding window and we will try to keep the library updated with better and more reliable approaches based on new research papers and our internal tests.

Apache-2.0 is correct one

Exactly! Uncertainty is critical to correctly evaluate LLM performance and we don't need reasoning models to spend thousands of tokens on simple questions

We want to integrate reasoning models as next steps because we see a lot of value in understanding better CoTs behaviour (DeepSeek R1 & Co)

Oh damn, you are right. It's my first opensource project and I didn't thought about it

In the example I'm using the instruction tuned version of Qwen2.5-7B to generate the insights