muds

Much of your argument rests on refuting the notion that the author feels "entitled" to a high-paying job. In that point, I agree with you. Any engineering undertaking is most productive when it is a meritocratic and competitive pursuit. People that feel "entitled" to an engineering job unfortunately need a reality check on their true competitiveness.

However, that doesn't seem like the authors core point. The authors' core point here is that they feel that the level of competition is past the point where their meritocratic achievements have any weight because to be competitive in the present marketplace, they need to either (1) inherently be _born_ in a different country with a low cost of living, (2) give up certain basic freedoms, (3) settle for a less skillful job where they can be an outlier in the distribution (for how long?) etc. -- all of which, to them, feel less meritocratic.

Of course, they might also feel "entitled" to a job, but that's not the interesting part of their argument (at least to me).

I agree that the data collection process wasn't ethical, and the professor should definitely be reprimanded for that. It's extremely sad that the coauthors weren't aware of this as well. And I feel terrible for the undergrads: their first research experience was publically rebuked for no fault of their own.

However, there is no shortage of projects with sketchy data collection methodologies on arXiv that haven't received this amount of attention. The point of putting stuff on arXiv _is_ that the paper will not pass / has not passed peer review in its current form! I might even call arXiv a safe space to publish ideas. We all benefit from this: a lot of interesting papers are only available on arxiv v.s. being shared between specific labs.

I'm concerned that this fiasco was enabled by this new paradigm in AI social media reporting, where a project's findings are amplified and all the degrees of uncertainty are repressed. And I'm honestly not sure how to best deal with this other than either amplifying the uncertainty and jankyness in the paper itself to an annoyingly noticeable level, or just going back to the old way of privately sharing ideas.

Maybe this is the best case scenario for these sorts of papers? They pushed a paper on a public journal, and got a public "peer review" of the paper. Turns out the community voted "strong reject;" and it also turns out that the stakes for public rejection are (uncomfortably, IMO) higher than for a normal rejection. Maybe this causes the researchers to only publically release better research, or (more likely) this causes the researchers to privately release all future papers.

Putting papers and code on arXiv shouldn't be punished. The incentive to do this is to protect your idea from getting scooped, and also to inform your close community on interesting problems that you're working on and get feedback. ArXiv is meant for work in progress ideas that won't necessarily stand the peer review process, but this isn't really acknowledged properly on social media. I highly doubt the Twitter storm would have been this intense if the twitter posts explicitly acknowledged this as a "Draft publication which hints as X." But I admit that pointing fingers at nobody in general and social media specifically is a pretty lazy solution.

The takeaway IMO seems to be to prepend the abstract with a clear disclaimer sentence conveying the uncertainty of the research in question. For instance, adding a clear "WORKING DRAFT: ..." in the abstract section.

I admit that Imagenet isn't the best analogy here. But I'm pretty confident that this data cleaning issue would be caught in peer review. The biggest issue which I still don't understand was the removal of the test set. That was bad practice on the authors' part.

I'm not sure what to make of this post. There is always a degree of uncertainty with the experimental design and it's not surprising that there are a couple of buggy questions. Imagenet (one of the most famous CV datases) at this point is known to have many such buggy answers. What is surprising is the hearsay that plays out on social media that blows the proportion of the results out of the water and leads to opinion pieces like these targeting the authors instead.

Most of the damning claims in the conclusion section (Obligatory: I haven't read the paper entirely, just skimmed it.) usually get ironed out in the final deadline run by the advisors anyway. I'm assuming this is a draft paper for the EMNLP deadline this coming Friday published on arxiv. So this paper hasn't even gone through the peer review process yet.

I've been struggling with keeping track of research experiments and code at the same time. This seems pretty cool! I like how this method is language agnostic and uses "matured" tools. Question: I'd love to give this a try; do you have any public code snippets?

The original comment, to me, reads more like "subject GRE is a definitive measure of eventual success in academia." I was arguing against the definitive part. Thanks for the clarification. Maybe it might be a good measure for your cohort, you, and people in similar situations.

> Almost no undergrad gets a first author paper

Maybe this is different in different fields but we have a lot of undergraduate first author papers in programming languages and machine learning. I mean -- through and through -- undergraduate students bringing up a topic, getting guidance from professors and senior PhD students, getting results by the end of the semester, and publishing results by next year. Even the people who end up "running the sds slides" either fall out by next year or end up working towards their own first author publications. I've always chalked this up to the experimental setup cost being very cheap in CS compared to in the "hard sciences" so most undergraduate students are already comfortable with all the tools they need to do research.

> I wasn't interested in trying to become a professor

I think this is precisely the variable that a standardized test cannot account for! I feel an "authentic" undergraduate research experience is successful if it helps students realize if research is right for them or not.

> ... papers ... the son of the department head...

I see where your frustration is stemming from. Sorry this was your first experience with undergraduate research.

Respectfully, I call bullshit. You can't quantify success in a PhD with a single variable. There are a billion ways research can go right or wrong -- irrespective of your personal pedigree. Your ideas might be too early or late for your community to grasp, maybe you appeal to the wrong audience, maybe you're unaware of an application of your research, maybe you don't have the right set of collaborators or need a perspective that, often times, emerges out of a lucky encounters with someone. I respect your experience but I don't want it to give people the wrong idea about research success...

> Paper authorship had NO correlation with research success

I think paper authorship demonstrates that you're willing to put in a non-trivial amount of work to persue a problem. That seems to be atleast one attractive skill in a PhD, wouldn't you agree?

> everything which makes programs useful is impure device access and state change, discretely sequenced over time

I haven't heard about this before actually. I'd love to hear more about this! "Impure," here, is PL terminology for functions that affect global state/arguments when you run them. right? So, brainstorming a bit, what this means is that making a diff. programming language that treats a NN module as a pure function won't actually be beneficial? I'm not sure if I'm drawing the correct conclusion but this is a really interesting point. Don't have an answer for this (yet!).

> grad. desc. et al. do not learn discrete constraints

Great Point! To push back a little on this. You're right that any discrete constraint will always mess up the smoothness of the function (eg: less-than-g is not smooth at x=g). However, we can engineer our way around this by relaxing a discrete constraint to its closest smooth approximation! So, we can implement the less-than-g function as a sigmoid that is shifted by +/-g. This introduces a parameter to control the slope of the sigmoid. In practice, I haven't had much difficulty learning programs even with a really steep slope for the sigmoid.

I don't see how nuclear is any more dangerous than any other renewable sources of electricity. The law of conversation of energy dictates that the residual waste energy has to go somewhere for any process. Any and all energy production mechanism are going to cause some disturbance in the natural order of things. It just turns out that the emissions due to "renewable" sources isn't something that does widespread ecological or societal damage. A hydroelectricity plant's "emissions" are in the form of riverbed erosion however, we can optimize around that by putting it in places which minimizes ecological and societal damage. For nuclear, we minimize this damage by placing the waste matter in large indestructible concrete bins.