suryabhupa
Joined 11 years ago
Karma
53
Submissions
20
Member since
Jun 12
Recent submissions
Comment · 2 years ago
In practice, and at scale, that's exactly what having <bos> and <eos> tokens allow you to easily and programmatically do.
Comment · 2 years ago
The announcements are live on Twitter! See this for example: https://x.com/suryabhupa/status/1806342617191379167
Comment · 2 years ago
Surya here from the core Gemma team -- we can think of a distillation loss as learning to model the entire distribution of tokens that are likely to follow the prefix thus far, instead of only the token in the training example. If you do some back of the envelope calculations, we can see that learning to model a larger distribution yields many more bits of information to learn from.
Comment · 5 years ago
This is really remarkable! How hard do you think it will be to support new models, i.e. does the tooling you’ve built generalize to you being able to serve other large scale models easily?
Comment · 8 years ago
Hi everyone! One of the creators of DFL here. In an attempt to more deeply understand fundamental concepts in machine learning, we designed Depth First Learning. It's a pedagogy for diving deep into machine learning by carefully tailoring a curriculum around a particular paper or concept and leading small, focused discussion groups. So far, we’ve created guides for InfoGAN, TRPO, AlphaGoZero, and DeepStack.
Since our launch, we’ve received very positive feedback from students and researchers. Now, we want to run new, online classes around the world.
We intimately understand that the process of curating a meaningful curriculum with reading materials, practice problems, and instructive discussion points can be very rewarding, but also time-consuming and difficult. We wanted to make sure that the people compiling the content understood that their efforts were well worth their time and consequently decided to launch a fellowship program.
Thanks to the generosity of Jane Street, we will provide 4 fellows with a $4000 grant each to build a 6 week curriculum and run weekly on-line discussions.
If you’d like to lead a class about an important paper in machine learning, please visit http://fellowship.depthfirstlearning.com to apply. We look forward to hearing from you, and I'm happy to answer any questions about it!
Comment · 9 years ago
Many machine learning and reinforcement learning models are susceptible to adversarial attacks; it's not unique to deep learning. However, because so many systems that are currently deployed in applications use deep learning, it's under particular scrutiny.
Comment · 9 years ago
There are talks of incorporating this into Excel at some point in the future, but it may take a _while_ before it can be fully productionized.
Comment · 9 years ago
That would be pretty cool to see what it learns, but I don't think we've tried that :P
Comment · 9 years ago
That's one manifestation of this kind of research being used in real life by programmers around the world. :)
Comment · 9 years ago
I'm not too familiar with evolutionary computation methods, but I imagine the approaches may be similar in nature.
Comment · 9 years ago
Theorem solving is very closely related to program induction (we just change the grammar). Just as with Python, the underlying search space would be incredibly large, and while in theory, we could simply change the DSL and it should work, it'll probably involve a few more iterations of the model or other insights to see this to fruition (but it's definitely not impossible).
Comment · 9 years ago
It turns out the full grammar of Python (and almost all real programming languages) is quite large; this is very early and new work in neural program synthesis, and so we chose a pretty limited DSL to make sure that we could at least solve this one before moving on to more general ones that contain state, conditionals, for-loops, etc. In theory however, we can apply the exact architecture to Python programs and see what happens. We haven't tried yet. :)