alexhutcheson

Glad to see the author making a serious effort to fill the gap in public documentation of RLHF theory and practice. The current state of the art seems to be primarily documented in arXiv papers, but each paper is more like a "diff" than a "snapshot" - you need to patch together the knowledge from many previous papers to understand the current state. It's extremely valuable to "snapshot" the current state of the art in a way that is easy to reference.

My friendly feedback on this work-in-progress: I believe it could benefit from more introductory material to establish motivations and set expectations for what is achievable with RLHF. In particular, I think it would be useful to situate RLHF in comparison with supervised fine-tuning (SFT), which readers are likely familiar with.

Stuff I'd cover (from the background of an RLHF user but non-specialist):

Advantages of RLHF over SFT:

- Tunes on the full generation (which is what you ultimately care about), not just token-by-token.

- Can tune on problems where there are many acceptable answers (or ways to word the answer), and you don't want to push the model into one specific series of tokens.

- Can incorporate negative feedback (e.g. don't generate this).

Disadvantages of RLHF over SFT:

- Regularization (KL or otherwise) puts an upper bound on how much impact RLHF can have on the model. Because of this, RLHF is almost never enough to get you "all the way there" by itself.

- Very sensitive to reward model quality, which can be hard to evaluate.

- Much more resource and time intensive.

Non-obvious practical considerations:

- How to evaluate quality? If you have a good measurement of quality, it's tempting to just incorporate it in your reward model. But you want to make sure you're able to measure "is this actually good for my final use-case", not just "does this score well on my reward model?".

- How prompt engineering interacts with fine-tuning (both SFT and RLHF). Often some iteration on the system prompt will make fine-tuning converge faster, and with higher quality. Conversely, attempting to tune on examples that don't include a task-specific prompt (surprisingly common) will often yield subpar results. This is a "boring" implementation detail that I don't normally see included in papers.

Excited to see where this goes, and thanks to the author for willingness to share a work in progress!

It depends if they are using a “vanilla” instruction-tuned model or are applying additional task-specific fine-tuning. Fine-tuning with data that doesn’t have misspellings can make the model “forget” how to handle them.

In general, fine-tuned models often fail to generalize well on inputs that aren’t very close to examples in the fine-tuning data set.

Are there any open source self-hostable tracking/project management tools that still have a committed team and forward momentum?

I used to self-host a Phabricator instance, which I liked a lot, but the upstream maintainer made the reasonable decision to step away.

My guess is there is not much of a niche for self-hosted solutions anymore. The GitHub Issues free tier covers most of the low-complexity use-cases, while higher-complexity use-cases are addressed by enterprise SaaS.

Boost.Heap has this functionality. Or if you want to stick with the standard library it’s fairly easy to use the *_heap functions from <algorithm> and just hand-code your own fix_heap(first, last, changed) function. Agree it would be more convenient to have it built-in, though.

Somewhat tangential, but did Nvidia ever confirm if they cancelled their project to develop custom cores implementing the ARM instruction set (Project Denver, and later Carmel)?

It’s interesting to me that they’ve settled on using standard Neoverse cores, when almost everything else is custom designed and tuned for the expected workloads.