bclavie

Hey! It’s more like comparing apples to apple pie.

BGE-M3 is a fine-tuned embedding models. This means that they’ve taken a base language model, which was trained for just language modeling, then applied further fine-tuning to make it useful for a given application, in this case, retrieval.

ModernBERT is one step back earlier in the pipeline: it’s the language model that application-specific models such as M3 build on.

They perform different roles, so they're not directly comparable.

Jina V3 is an embedding model, so it's a base model, further fine-tuned specifically for embedding-ish tasks (retrieval, similarity...). This is what we call "downstream" models/applications.

ModernBERT is a base model & architecture. It's not supposed to be out of the box, but fine-tuned for other use-cases, serving as their backbone. In theory (and, given early signal, most likely in practice too), it'll make for really good downstream embeddings once people build on top of it!

We had a bit of a discussion around it, but I figured that 6 years warranted the prefix, and it's easier to remember in the sea of new acronyms popping up everyday.

Besides, PostModernBERT will be there for us for the next generational jump.

Thank you! We're fixing the link.

Hey, Ben here, one of the paper's core author authors. The responses you got were mostly spot on.

For (1), it's because BERT has both noticeably fewer parameters, and we're comparing at short context length (in the interest of providing a broader comparison), so local attention is a lot impactful than it is at the longer context lengths.

For (2), most LLMs are actually decoder-only, so there is no "encoder" here. But also, there's not a lot of LLMs in the ±100M parameter range in the first place!

Hey! Thanks for posting this. I'm the author of this post -- please feel free to shout if you've got any questions

Thanks! I agree -- I find it much easier to skim a few paragraphs than to skim through a video when trying to consume information quickly if I'm not sure I want to commit to a full, long vid. Hoping to make it useful enough that it ends up paying for its own server costs so I can keep it around!

Merci! It's early on but I'm quite happy with how the first prototype turned out.

> How do you break down the segments/sections? Is it just fixed time? What happens if there are more than one topic discussed in the segment?

Currently it's just a dumb fixed time rule, based on max video length (3 or 5mn segments). I played around a bit and it's the easiest way to implement things that works remarkably well. If there are multiple topics, there a few branching paths in the code, but a lot of it comes down to believing in the LLM's ability to make sense of it. I've got some ideas to improve, but would need a bunch of work to implement well.

> Are you using both chatgpt and mistral? Do you use them for different tasks?

There's a degree of A/B testing (well, "A/B testing", since we're not collecting feedback) where some of the summaries are GPT, some of them are mistral, mixed together for the same video. Mistral being superbly fast means it's also really useful to support the branching coding logic (e.g. something I'm working on right now is having an entirely different summarisation style if a video is about sports, and while a logistic regression would do that pretty well, it's not particularly robust, and won't tell me what sport it is if the transcript is full of typo) or to clean up the video transcripts.

Haha merci! Prends ton temps, pour le moment le tout tourne sur des crédits cloud gratuits alors la seule utilité du bouton premium c'est de faire de la lumière. Bonne chance pour ta recherche!

> Impressive, thanks. How could one run something like that on local videos ?

It depends how involved you'd want it to be really. You can get a very simple summary using something like Whisper to transcribe a video and having basic LLM calls. More involved summaries/segment breakdown/fine-tuned models would be a lot more work, but might not be needed for something quick with local vids?

> Btw I love the name.

Thank you! This is actually a project I had back in 2018ish, which fizzled out because I didn't have enough time to get good enough summaries going during the pre-LLM era. I let the domain name expire and a few weeks ago realised it was still free, so got building again and re-bought it!