nimitkalra
Joined 11 years ago
Karma
349
Submissions
35
Member since
Aug 14
Recent submissions
Comment · 1 year ago
One method to get a better estimate is to extract the token log-probabilities of "YES" and "NO" from the final logits of the LLM and take a weighted sum [1] [2]. If the LLM is calibrated for your task, there should be roughly a ~50% chance of sampling YES (1) and ~50% chance of NO (0) — yielding 0.5.
But generally you wouldn't use a binary outcome when you can have samples that are 50/50 pass/fail. Better to use a discrete scale of 1..3 or 1..5 and specify exactly what makes a sample a 2/5 vs a 4/5, for example
You are correct to question the weaknesses of a panel. This class of methods depends on diversity through high-temperature sampling, which can lead to spurious YES/NO responses that don't generalize well and are effectively noise.
[1]: https://arxiv.org/abs/2303.16634 [2]: https://verdict.haizelabs.com/docs/concept/extractor/#token-...
Comment · 1 year ago
There are technical quirks that make LLM judges particularly high variance, sensitive to artifacts in the prompt, and positively/negatively-skewed, as opposed to the subjectivity of human judges. These largely arise from their training distribution and post-training, and can be contained with careful calibration.
Comment · 1 year ago
Some other known distributional biases include self-preference bias (gpt-4o prefers gpt-4o generations over claude generations for eg) and structured output/JSON-mode bias [1]. Interestingly, some models have a more positive/negative-skew than others as well. This library [2] also provides some methods for calibrating/stabilizing them.
[1]: https://verdict.haizelabs.com/docs/cookbook/distributional-b... [2]: https://github.com/haizelabs/verdict
Comment · 10 years ago
In order to see download counts for specific packages, you'll need to navigate to:
bintray.com/homebrew/bottles/{PACKAGE}/view#statistics
Comment · 10 years ago
From what I've noticed, these "transpilers" output code that is readable (the code itself is written as though a human wrote it) where as "compilers" output code that has been optimized and show effects of name mangling in the code itself, etc. Just an observation.
I think it makes sense to use a different term for this "compiler"-esque behavior. For example, I might edit the output of CoffeeScript generated Javascript whereas I wouldn't know how to modify the output of gcc.
Comment · 11 years ago
https://nimit.io