ivo-42

Joined 3 years ago

Karma

74

Submissions

5

Member since

Apr 4

pre-training data @ aleph alpha

https://github.com/ivo-1

View on Hacker News

Recent submissions

Comment · 28 days ago

Stay tuned, this is just the beginning. Merging with Cohere will help in scaling, too.

Comment · 28 days ago

Hey, I worked on pre-training data Kolibri. We spent considerable time and effort to go beyond just translating. For example by building a pipeline that processes Common Crawl dumps specifically for German. You might be interested in a related blog post: https://aleph-alpha.com/en/blog/sauerkraut-not-burgers-why-g...

We also talk more about German pre-training data in the tech report: https://aleph-alpha.com/downloads/tech-report.pdf section 2.3.2.2.

Comment · 28 days ago

By various forms of deduplication (exact, fuzzy, substring), heuristic filters and distilling quality classifiers that annotate our data. Synthetic rephrases can also be considered a form of cleaning/getting more out of existing noisy data.

We have a lot of details in the tech report if you want to go deeper.

Comment · 28 days ago

I worked on Kolibri, in particular pre-training data and mid-training. We strive to be as open as possible. Glad you like it.

Comment · 28 days ago

Hey there, I worked on Kolibri pre-training data. Agree with the point that non-US/non-Chinese labs need to pool effort. The merger with Cohere is public, nothing to hide and I'm also excited about it personally.