MurizS

Joined 8 years ago

Karma
673
Submissions
63
Member since
Aug 21
Training multimodal models. Here goes nothing:

https://hesiod.ch/

View on Hacker News

Recent submissions

Comment · 10 months ago
Native since v146, about:config -> browser.tabs.splitView.enabled
Comment · 1 year ago
Which in turn lets the people at Moonshot AI worry about that for them, the only provider for this model as of now.
Comment · 1 year ago
Comment · 1 year ago
I think GP was probably referring to "Scaling Data-Constrained Language Models" (2305.16264) from NeurIPS 2023, which looked first at how to optimally scale LLMs when training data is limited. There is a short section on mixing code (Python) into the training data and the effect this has on performance on e.g. natural language tasks. One of their findings was that training data can be up to 50% code without actually degrading performance, and in some cases (benchmarks like bAbI and WebNLG) with improvements (probably because these tasks have an emphasis on what they call "long-range state tracking capabilities").

For reference: In the Llama 3 technical report (2407.21783), they mention that they ended up using 17% code tokens in their training data.

Comment · 1 year ago
Most likely "The Brothers Karamazov".
Comment · 2 years ago

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton