nighthawk454

At the risk of oversimplifying, it seems the field looked at neuron activations and couldn't find the pattern, so a series of convoluted hypothesis were proposed to explain it away. When the actual answer was the pattern was very simple they just missed it. Little bit embarrassing really.

The good news I guess is the actual patterns make a lot of sense with what you'd expect the neurons to do, and don't require nearly as much 'black magic who knows what wonders scale can create'.

which amounts to nearest-neighbor

Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.

Somewhat unclear how particularly novel this is vs a way to save compute.

Ah good point, no udp. I was looking at webtorrent but it does share the same limitation, unless you’re on the nodejs (non-browser) version.

Looking forward to trying bitbang! Seems super neat

> it still needs to bootstrap the connection via central server. I think that always might be a limitation of P2P

I think Iroh has a way of doing it using either DNS or BitTorrent DHT. Not perfect but fairly decentralized

This tracks with what I’ve heard around as well. What changed in the financial instruments?

My understanding is a lot of the loans have gone PIK or otherwise essentially aren’t serviceable at current prices. Do you think that’s resolvable somehow or just lagging implosion?

Some potentially related stuff on the topic:

Anisotropy in word embeddings dates back to at least 2017 with word2vec - where there were zero layers.

The cone-shaped anisotropy in transformers is known since at least Gao et al. 2019. That lineage explained it fairly intuitively as an artifact of word frequency and softmax geometry (so a training dynamic).

A variety of papers followed up by adding post-hoc ‘whitening’ steps (from classical statistics/NLP), then adding regularizers to the loss to penalize the anisotropy, eventually penalizing the covariance matrix (a la VICReg), and then the SIGReg method as a computationally much cheaper way to approximate the full covariance.

As another commenter pointed out it’s also similar to the InfoNCE/contrastive learning objectives. Where terms to increase uniformity (spread out evenly) on the hyper sphere were added. Like the SimCSE (Gao 2021) paper or the excellent alignment/uniformity breakdown from Wang & Isola 2020.

This proposed dispersion loss seems to be similar in that it pushes things apart by penalizing cosine similarity. Although this one works on the tokens within one sequence. Usually contrastive methods mean pool the sequences and then contrast against the other pooled sequences in the batch.

Ah, I hadn’t thought about the generational aspect that’s interesting. The aesthetic totally makes sense to me when the music is intended for it / designed with it in mind, which I guess quite a lot of music is.

I particularly dislike when old intentionally-dynamic music is remastered to be “modernized” into a brick, which is sort of the opposite direction.

> Cinema mixes

I didn’t know about these, that’s neat! Makes sense that the levels can’t really be the same in my living room as a theater. Is it really a whole separate mix or just some compression in mastering?

I really hope that’s not another masterings collection rabbit hole I’m about to fall down haha. I’ll look out for some Dolby certified venues in my area too

> 1 … they will often chose the more compressed/louder one

I’ve always been curious - but presumably that’s true even after volume matching?

> 3 Compressed sound works better in noisy environments and as background music

I’ve heard this is also why film and video game soundtracks are often very compressed, even when orchestral, because they have to fit in the background with dialog/sfx

ReplayGain is nice - but note it doesn’t ’fix the compression’. Compression and dynamic range is about loud/quiet _within_ the track. ReplayGain just turns the volume up and down for the entire track, the point being so all your tracks play back at about the same level. It saves a preset on the volume knob for you essentially.

If you remember making a playlist where one song is suddenly much louder than the last, and you’re riding the volume knob on every other song, you’ll see why this is nice!

It’s completely true, when the vinyl has a different mastering. It can be a completely different version. It’s not because it’s vinyl