canyon289
Joined 10 years ago
Karma
1,490
Submissions
136
Member since
Oct 11
Recent submissions
Comment · 2 months ago
A lot of different ways. Every guide on this guidebook runs on a ~$500 gpu or a couple of dollars on a cloud instance with a bigger GPU
https://ravinkumar.com/GenAiGuidebook/book_intro.html
This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware.
If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi.
https://developers.googleblog.com/en/introducing-gemma-3-270...
The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here.
And FWIW I'm a staff researcher at Deepmind (opinions here are my own) so I want to specifically encourage all people out there, you can learn a lot about how these LLMs work at home, for (mostly free), using resources like colabs or spot pricing on accelerator providers. There's many great resources out there and I encourage anyone willing to learn to go for it!
Comment · 2 months ago
Disclaimer, I work on Gemma and open models at Deepmind and the opinions here are my own
There were open models from EleutherAI (GPT-Neo), Google Brain (T5X, Bert), and HuggingFace was promoting open models (and others doing open work I haven't listed here) all prior to 2023 and the big Chatgpt moment.
https://github.com/EleutherAI/gpt-neo/releases
https://github.com/google-research/bert
https://github.com/google-research/t5x
If you're learning about AI models it's still worthwhile to review these models and codebases because they continue to be the basis of the technology that's being produced today! It'll give you a good perspective of how things have changed, similar to say learning about propeller planes before moving onto modern jet engines.
Comment · 3 months ago
Join the waitlist! It's in the blogpost
Comment · 3 months ago
Totally fair and thank you for nit picking! I edited my comment. AI2 is doing amazing stuff, I read their papers and code all the time and appreciate how open they are.
Comment · 3 months ago
I'm a researcher at Deepmind that contributed to these models. (And the opinions here are my own)
Just want to say, Deepmind is a great place to work and the only (Edit: one the few unique labs!) lab where you can move from large frontier models (Gemini), frontier open models (Gemma), robotics (what you see here), science (weather, biology, more) and basically any other topic related to intelligence. It's really an incredible place to be, with incredible people. Consider joining! And thank you for the enthusiasm here.
Comment · 6 months ago
This is a nice writeup!
Comment · 7 months ago
This is going to sound like a corp answer but I mean this genuinely as an individual engineer. Google is a leader in its field and that means we get to chart our own path and do what is best for research and for users.
I personally strive to build software and models provides provides the best and most usable experience for lots of people. I did this before I joined google with open source, and my writing on "old school" generative models, and I'm lucky that I get to this at Google in the current LLM era.
Comment · 7 months ago
I dont have the metrics off hand, but I'd say try it and see if you're impressed! What matters at the end of the day is if its useful for your use cases and only you'll be able to assess that!
Comment · 7 months ago
On this one I dont know :) I'll ask my friends on the evaluation side of things how they do this
Comment · 7 months ago
Its hard to say because Pixel comes prepacked with a lot of models, not just ones that that are text output models.
With the caveat that I'm not on the pixel team and I'm not building _all_ the models that are on google's devices, its evident there are many models that support the Android experience. For example the one mentioned here
https://store.google.com/us/magazine/magic-editor?hl=en-US&p...
Comment · 7 months ago
We are always figuring out what parameter size makes sense.
The decision is always a mix between how good we can make the models from a technical aspect, with how good they need to be to make all of you super excited to use them. And its a bit of a challenge what is an ever changing ecosystem.
I'm personally curious is there a certain parameter size you're looking for?
Comment · 7 months ago
You could try Gemma4 :D
Comment · 7 months ago
Hi all!
I work on the Gemma team, one of many as this one was a bigger effort given it was a mainline release. Happy to answer whatever questions I can
Comment · 8 months ago
I work on Gemma and Gemini models I want to echo Daniel's point here. Small finetuned models have their place even with larger general purpose models.
For example last year with Daniel/Unsloth's help we released a tiny specialized model that can get equivalent to Gemini level purpose specifically for FC. For folks that need efficient limited purpose models small models like this can fit a specific need.
https://blog.google/innovation-and-ai/technology/developers-...
Especially on device. https://developers.googleblog.com/on-device-function-calling...
It's the same with chips, we have general purpose CPUs but we still have specialized silicon for tasks that are smaller, more power efficient, cheaper, and because they're single purpose it simplifies and derisks certain designs.
And I have to add, if you want to learn about finetuning models efficiently the Unsloth guides are at the top of my list. They're practical, have all the technical details, and most importantly Daniel and the others are working around the clock to keep it up to date in what is an incredibly fast moving space of models and hardware. I am continually astounded by their work.
Comment · 10 months ago
I'm with you! Small generative models are awesome, I thought so a decade ago and I still think so now! The size of what is "small" has definitely increased though, I used to think a 100 parameter model was large back in 2016, but here I am now saying 270 million is small :)
https://ravinkumar.com/