parsimo2010

I also look at the ECI from Epoch.ai as well, but they haven’t evaluated Carbon yet. Probably because the model isn’t widely available yet.

We’re literally talking about the same thing, but ascribing different motivations.

I do think the test a new model and realize their index is broken- the new model isn’t on top, so they change the weights to make sure it’s on top.

The difference is that you seem to think they are doing so out of some innocent feeling that the new model is actually better, while I think they are doing so because they were paid to do so. I think my interpretation is more likely, but that’s just me.

Just so everyone is aware of the "coincidences" that happen with this index:

- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.

- Today the newest model announced from a big lab tops the charts.

I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.

It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.

Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.

If Borcherds accepts TAs then they work for him. The point was that he can do practically whatever he wants and doesn't have to worry about it harming him.

Thanks for explaining to me how the academic system I work in functions. Just like most managers don't literally pay their employees, the company does. The university pays TAs through funds from their various revenue streams, including endowment, tuition, grants to the university, and through indirect costs collected from research grants to researchers, probably including Borcherds own grants. So really a lot of different things pay the TAs, the university is just the clearing house.

Again, I’m not making a moral statement. But you yourself basically just said grading is a menial task that is below the station of a professor. (BTW, I teach at a small school and do all my own grading each semester for between 45-80 students depending on the courses I’m teaching that semester)

So if grading is part of the development process for a graduate student, then go ahead and continue. But that’s not the claim in the article. And if grading doesn’t serve a purpose anymore, then stop doing it. You can pay your assistants to do nothing for 20 hrs/week, or you can find literally anything else for them to do.

I don’t know Borcherds, so this isn’t intended to sound mean to him, but he could pay his TAs or RAs to do hula hoops naked in public for 40 hours a week and it wouldn’t affect him. He’s a high profile mathematician and the worst thing is that next year’s grant money is a little less and he can’t pay for as much naked hula hooping spectacle.

Entire math departments at multiple lower profile schools will close their doors before he has any actual concern about his future as an academic.

But your point is correct for other professors, much less secure in their positions, who have similar problems with AI and actually do have to concern themselves about using their TAs and RAs productively.

The irony here is perhaps missed by many. He didn’t grade homework in the first place. He paid assistants to grade. This is common practice among tenured professors with grant money to spend and graduate students beholden to them (I’m not making a moral statement, just noting this is common practice).

Aside from the irony that Borcherds probably hasn’t graded a homework set in a while, instead let’s examine the purpose of homework. It helps you learn. You can read a chapter and think you understand the material, but doing homework makes you engage more with the material. Recently we’ve used the external motivator of course points to encourage this good habit. But this turns the relationship between teacher and student into a transaction- you do a problem, you get a point. And most young students really don’t care beyond that transaction. They are going to college to increase their earning potential, and that’s about it.

So if we’re not going to rethink the purely transactional relationship schools have with most students, then you probably should stop giving points and grading homework. You have to figure out a new transactional structure that can’t have your tuition made irrelevant by a $20 subscription. That’s possible to figure out. A lot of people have pointed out various equity risks but it’s really just a matter of balancing the AI risk with the equity risk at the weighting acceptable to your institution.

You could also attempt the much more challenging prospect of turning the transactional nature of college into some other relationship which fosters a student’s intrinsic motivation of learning for the sake of learning, but I bet your institution goes bankrupt in the process. So really you should just stop grading homework and put 100% of the grade weight into proctored assessments with some sort of firewall against AI tools. Anyone that faces an equity disadvantage because of this was probably also disadvantaged by the fact that you let rich kids with premium AI subscriptions get better grades on their homework.

A sizable part of the country thinks it’s okay to regulate firearms. Under your logic, people should be responsible for who they shoot, don’t blame the gun when a human pulled the trigger. While gun control advocates have that pesky second amendment hampering their efforts, AI/LLM systems have no such protection.

Regulatory capture is certainly something to be wary of, and obviously Anthropic is only going to be in favor of regulation that helps them in some way. But some form of regulation is coming. There is nothing that can stop it, unless you somehow make more money than these guys do and fund a massive counter-lobby. Spoiler alert: the only guys that have more money than Anthropic and OpenAI have all invested in them.

While I understand your thinking, he rolled Twitter into SpaceX. He absolutely uses it as a financial vehicle, regardless of how legitimate the space business is.

Not sure if this has been noticed, but the pricing is “$299 (U.S.) or $49.83 (U.S.) per month for six months,” which means paying in installments is 2 cents cheaper than paying the whole price upfront. 49.83*6=298.98. So you’re not just getting a zero interest loan, you’re getting a discount.

I’m not advocating for the practice of chaining kids to a desk…

I’m trying to point out that a tool might be helpful for you to practice, but if it isn’t used then the tool can’t help. Maybe if you were forced to use the tool then you could see the benefit, but as you noticed, the act of “forcing” something may feel undesirable from a moral perspective.

I think this is what makes this so ridiculous. They put “room temperature” in the title and while it might not have been intentionally nefarious, it definitely has the effect that most readers assume it is about superconductors. There is nothing special about a room temperature semi conductor, your CPU, RAM, and storage drives have worked at room temperature for decades…

And if not for that word, the article would be completely unexciting. You found that you can make magnets out of multiple materials? We already knew that. If the materials aren’t abundant/cheap and easier to manufacture then this isn’t a story. And neither of those claims are tested or verified in this. So this isn’t a big deal.

A very honest take. They tried something, it showed a very small improvement, and they acknowledged it. Acknowledging that the cause appears to be a lack of engagement agrees with my own experience. You can give students great tools but that doesn’t make them interested in math. Khanmigo might be a good tutor but it won’t help unless you’re chaining kids to a desk and not letting them leave until they finish some quota of problems.

You could always curl the install script, and modify it to run the virus scan in between the build and install steps.

One reason is that the llama.cpp team (GGML) has strict requirements that a human must understand the code they are contributing. If a project is fully vibe coded they can’t contribute. So a lot of projects where an AI went and coded a bunch of custom kernels to increase speed are left to their own devices.

I think this is a fine behavior. We can have upstream purists that are strict gatekeepers but don’t get in the way of downstream forks. Debian has some this in the Linux landscape for a long time, and it has enabled Ubuntu, Mint, etc. to flourish without compromising themselves.