seamossfet

Coding and math are easily verifiable, language and prose is not.

As a result, I don't think LLMs will reach this state easily.

>Huh? You simply don't turn on the TV when the kid is there. If you want to save even more money and energy, don't get a TV.

If there is a TV at a restaurant and the kid sees it, do you tell them to turn it off for you?

If you walk by the electronics section at a walmart do you demand the workers turn off all the TVs before you pass by?

Do you scout every location prior to visiting to make sure you avoid TVs at all costs?

My point is the framing of kid cannot even _hear_ the TV without ruining the child is ridiculous.

Again, having the TV on to watch the news in the morning is not the same thing as plopping the child in front of cocomelon and letting them brainrot in the morning.

I'm just not convinced watching the news in the morning on the TV with our kid in the other room eating breakfast is going to ruin my child.

The way the post is framed, it makes it seem like you have to completely remove all TVs from the child's life to be consistent with the medical literature which doesn't seem correct.

>Most people use password123 as password, and like using the same password on every site, so saying that's bad is a bit silly?

That's a ridiculous false equivalence and you know it.

Choosing a poor password is nowhere near analogous to the investment and energy requirement of creating completely TV-free spaces for children.

I'm questioning whether that degree of separating is consistent with the medical literature, not whether we should "do something bad because we've always done something bad"

From the perspective of other people, most parents will almost always be considered a bad parent.

Parenting is hard, and strangers are always brutal judges of you.

Best not to concern yourself with the opinions of strangers.

Do keep in mind that the vast majority of people posting in this thread of grown up with TVs in the background. The 0-3 year rule is a little silly.

Like sure, of course, don't let the kid just mindlessly watch TV. But removing the TV entirely from all spaces they could possibly be in is too extreme and the benefits of that are not clear other than expressing the parents' own maniacal anxiety which (in of itself) is not a good thing to teach the child.

I agree with the framing but the claim is too strong.

I don't think Grassmannian manifolds are necessarily a better representation than simple query/key inner products. It's a valid solution, and it does have advantages in some cases.

The Grassmannian is only preserving the subspace orientation, so we lose information about the actual linear map and basically throw out its basis.

Like everything, this is a trade off. You're trading information for a lower rank representation.

I'm still reading through the paper on the implementation side, but the other caveat I'd be wary of is compute. Does this projection add MACs to the overall model which would make inference slightly slower even with lower rank representations? I don't know. Like with any technique that trades constrained information representation for space/compute you _are_ widening your search space during training. I'd be curious what the impact of this is on relative data needed to train a model using this architecture versus "regular" attention.

Clever, I wonder what would happen if you did this in reverse? Like, if we froze the weights then added some layers trained on a bunch of weird edge cases then tried to convert those weights back into branches.

QA through entropy lol

A byproduct of working remote is the amount of human connection in general has gone way down. The only time I "see" the people I work with anymore is when we're on a meeting.

Meetings that could have been an email, in some ways, are replacing hallway and water cooler conversations. Are they a good replacement for that? Not really.

For those saying this creates a threat to Nvidia's deathgrip on the market, it really doesn't

The main reason people don't use Huawei hardware is due to regulation. If you're in the US you'd straight up get your hardware seized by customs because they're illegal to buy or bring into the country. EU has similar guidelines I believe.

Nvidia will continue to have its monopoly as long as the powers that be mandate it.

Yeah, there's a tradeoff. pre-training with QAT needs more data and a ton of hardware, so it's easier to just take an existing open weight model and quantize it. This works, but it'll underperform a model that had quantization as a training target.

Matters more for 1-bit and 2-bit models.

>I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality

There's literature on this actually, if you want to get that low the model really should have quantization as a pretraining target otherwise larger models that are quantized after collapse.

In fact, to do this effectively, you need somewhere near 50x chinchilla to do QAT on super tiny targets like 1-bit or even 2-bit. These large models already need an astronomical amount of training data that doing proper QAT that small isn't really feasible unless there are breakthroughs in the architecture.

4-bit models and smaller _can_ perform well but not through just naively quantizing the existing weights of a large model.

How else would they get their marketing stories unless the agents can "break out" of containment?

>You literally ask the agent to do something and it does it

It's not that simple, you have to remember at the end of the day these things are just doing next token prediction. If you don't give it the proper tokens to attend to, then your outputs won't be satisfactory.

You can get stellar outputs from LLMs, but it really is a function of how well you manage your input tokens.