dontwearitout

They certainly can, see https://en.wikipedia.org/wiki/Resonance_escape_probability. The all-important number in nuclear reactions is "k", the average number of child neutrons a single neutron will produce. The neutron population follows an equation something like N = exp((k - 1) * t). For k<1, you get exponential decay, and for k>1, you get exponential growth (until everything becomes a plasma and k changes). Check out https://en.wikipedia.org/wiki/Six_factor_formula.

Comment · 2 years ago

Claude notably does not use RLHF, but uses RLAIF, using a LLM to generate the preferences based a "constitution" instead of human preferences. It's remarkable that it can bootstrap itself up to such high quality. See https://arxiv.org/pdf/2212.08073 for more.

Comment · 2 years ago

Are dendritic sub-compartments necessary to explicitly model, or does this work just imply that biological neurons are complicated and are better modeled as a multi-layered artificial network, rather than a single simple computational unit?

Similarly, do you think that spiking networks are important, or just a specific mechanism used in the brain to transmit information, which dense (or sparse) vectors of floats do in artificial neural networks?

Comment · 2 years ago

Your "blip" anecdote is super interesting. I know that exact sensation of disorientation (but without the migraines thankfully), and I'm curious if this has been studied in depth anywhere.

Comment · 2 years ago

I don't know if you're out of the loop but the AI scene has dramatically changed in the past few years. We are solidly out of the AI winter.

Comment · 2 years ago

I haven't heard the term "shallow basin hypothesis" but I know what it refers to, these two papers spring to mind for me:

1) Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://arxiv.org/abs/1802.10026

2) Visualizing the Loss Landscape of Neural Nets https://arxiv.org/abs/1712.09913

There's also a very interesting body of work on merging trained models, such as by interpolating between points in weight space, which relates to the concept of "basins" of similar solutions. Skim the intro of this if you're interested in learning more: https://arxiv.org/abs/2211.08403

Comment · 2 years ago

I don't think a dot product between high dimensional vectors is considered a convolution? I'm familiar with convolution between continuous functions, and with kernels in neural networks providing invariance. I'd love to learn more if you have any links that expand on your statement.

Comment · 2 years ago

Lots of interesting work out there; time will tell!

Comment · 2 years ago

The transformer module is currently dominating ML, and is widely used in text, vision, audio, and video models. It was introduced in 2017 and shows no real signs of being displaced. It has no convolutions.

https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... http://jalammar.github.io/illustrated-transformer/