avdelazeri

I wish you would provide more information. About time, costs, reliability (how many duds did you have? did you need to do any babysitting or etc) and so on But it is a very impressive looking demo, for sure.

Mainstream tends to clown on Jumper, but I had a lot of fun watching it with my pals back then.

Similarly, I'm a grad school student studying AI (well, computer vision). When I called my internet provider to complain and ask for maintenance (a fool with a tall truck ran on our street and tore some of the cables) I could only get annoying AIs to talk with me. They kept arguing that my service was working properly despite the snapped cables. I reckon that the AI assistants were hallucinating the answer to a ping or something like that. I found another provider who actually gave me the option to talk with an actual human being and switched in the same day.

One study about faculty hiring people they know, and the other about high school students cheating on assignments... What was the original claim again?

Given that their publication says the dataset is freely available on Huggingface that's at least something ig

Now, I don't know anything about neuroscience or brain development, but hopefully I can explain the statistics in a way useful to you.

Imagine there are two groups A and B. One group, A, has slower reactions on average and high average activity The other, Group B, has higher reactions and lower than the Group A's activity. Yet inside both groups the general trend is that if someone is slower than the average reaction of their group then they're also below the average activity for their group.

If we look at the overall means without distinguishing groups, slower reaction is correlated positively with higher activity (kids from group A have higher activity and slower reaction in general, which pushes the correlation upwards. As long as the relationship in Group B isn't too strong the upward trend from Group A can easily dominate overall correlation) but inside each group the trend is actually the opposite.

This applies pretty much every time you're comparing samples. If I understood your quote correctly, they're studying a child's reaction time vs activity level by comparing the same kid in different times. The same logic applies, a person can exhibit the opposite trend to the populational average due to the same mechanism above. This can be even more dramatic, because once you start looking at averages you start losing time dependency information.

More broadly (and more formally), multivariate covariance splits in within-group and between-group terms, so if the signs of the terms are different the magnitude of one can dominate the overall sum and flip the sign.

I don't think this is the slam dunk you think this is. LinkedIn's existence is, in fact, a net negative for the human race.

We must know, we will know.

OGame and Travian are two names that really take me back. Those and Tribal Wars, I played them a lot back when I was a teenager.

Has Claude stopped claiming to be deepseek when prompted in Chinese yet? It wasn't long that it hit the news and blogs

Afaik turning up the temperature slowly wouldn't work on an actual frog. But works on people without fail.

Lack of will. That was one of the main results from the survey from Whitaker in 2020. Making your code reusable and easy to understand is significant work that had no direct benefits for a researcher's career. Particularly because research code grows wildly as researchers keep trying thungs.

Working on the next paper is seem as the better choice.

Moreover if your code is easy for others to run then you're likely to be hit with people wanting support, or even open yourself to the risk of someone finding errors in your code (the survey's result, not my own beliefs).

There are other issues, of course. Just running the code doesn't mean something is replicable. Science is replicated when studies are repeated independently by many teams.

There are many other failure modes SOTA-hacking, benchmarking, and lack of rigorous analysis of results, for example. And that's ignoring data leakage or other more silly mistakes (that still happen in published work! In work published in very good venues even)

Authors don't do much of anything to disabuse readers that they didn't simply get really look with their pseudorandom number generators during initialization, shuffling, etc. As long as it beats SOTA who cares if it is actually a meaningful improvement? Of course doing multiple runs with a decent bootstrap to get some estimation of the average behavior os often really expensive and really slow, and deadlines are always so tight. There is also the matter that the field converged on a experimentation methodology that isn't actually correct. Once you start reusing test sets your experiments stop being approximations of a random sampling process and you quickly find yourself outside of the grantees provided by statistical theory (this is a similar sort of mistake as the one scientists in other fields do when interpreting p-values). There be dragons out there and statistical demons might come to eat your heart or your network could converge to an implementation of nethack.

Scale also plays into that, of course, and use of private data as the other comment mentioned.

Ultimately Machine Learning research is just too competitive and moves too fast. There are tens of thousands (hundreds maybe?) of people all working on closely related problems, all rushing to publish their results before someone else published something that overlaps too much with their own work. Nobody is going to be as careful as they should, because they can't afford to. It's more profitable to carefully find the minimal publishable amount of work and do that, splitting a result into several small papers you can pump every few months. The first thing that tends to get sacrificed during that process is reliability.

While I never measured it, this aligns with my own experiences.

It's better to have very shallow conversations where you keep regenerating outputs aggressively, only picking the best results. Asking for fixes, restructuring or elaborations on generated content has fast diminishing returns. And once it made a mistake (or hallucinated) it will not stop erring even if you provide evidence that it is wrong, LLMs just commit to certain things very strongly.

Regulatory capture is an ugly thing.