martincsweiss

Author here, and Postdoc with Hugo Larochelle and Chris Pal.

I saw the LocalWiki last week, and it inspired me to make a learning tool based on "local" research papers. Many ML researchers write code that relies on a long history of research in ML and applied math. But which papers are you actually building on?

You can paste a GitHub repo and Standing on will read its Python code match pieces of it back to the research papers from which it arose.

nanoGPT, for example has 15 Python files and links to 13 papers including Adam, AdamW, LayerNorm, GELU, FlashAttention, etc. If you go back further, you get Adam and then AdaGrad (2011), Zinkevich's online convex programming (2003) and finally you wind up with Nash (1950) and Kakutani's fixed-point theorem (1941).

It's a very simple symbol table map with 4,800 classes and functions in torch, transformers, diffusers, timm, sklearn and others to the papers they implement. I used Claude and Codex to label the symbols and named each paper's parents, and check them against our ML research map, Lacuna.

We only support python and machine learning papers only for now. You can export to BibTeX, CSV and Markdown.

Yes I’d say so. That and a line of work on compositional generalization in deep learning.

I don’t think that’s where we are headed. Humans are the bottleneck but they aren’t replaceable by LLMs. We experience high fidelity multimodal observations all day long, generated by a 13 billion year old gigantic quantum computer. LLMs get a few quantized tokens.

The universe has too much detail for us not to contribute useful information to a human-so scientific process

Author here. I work on AI tools for ML researchers, and have had this essay rattling around in me for a while (since I completed my PhD last year, actually).

Seeing OpenAI's publication on "Working with academic communities to build scientific understanding" motivated me to finally post it.

I expect we’ll continue getting faster at producing results than at turning them into ideas other researchers can understand and use.

Academia already rewards new discoveries more than the work explaining and connecting them (I talk about this at length, and am working on a subsequent piece that goes deeper into the history of that).

OpenAI organizing groups to understand these results seems like a useful response, but I think that structurally the incentives and culture aren't there yet.

We actually have a high-risk research idea for how to do this. It involves looking at the clustering as a vocabulary that compresses the concept space, and (one form) of significance as how much your paper / addition of concepts changes the clustering/vocabulary. I.e., if you change the optimal compression of the space, that's probably important.

The academic paper cost a few thousand bucks, max $10k in compute.

Lacuna is far more expensive and ~100x larger. If you knew what I know and started building it today (LLMs got cheaper).

Paper summaries cost ~$50k, concept extraction cost ~$25k, author profiles cost $25k, research proposals cost another $50k. So call it $150k for model calls.

It's also an online / updating daily so that costs too, plus infra, plus "spillage".

I had not, and I find it deeply hilaritragic that his paper is paywalled by JSTOR. Thinking of you, Aaron Swartz.

Hmm, interesting. Yea I think ML is a much smaller field, too. ArXiv + all the conference / applied ml stuff is ~150-250k papers per year. PubMed is 1.5-1.7 million new records per year, and like 40,000,000 total records, whereas in ML we only had to process a couple million. Plus, pretty much all of ML is open-access, whereas medical is ~50% closed.

If somebody gave us a $10-$20m grant for compute, and a few (dozen) helpful doctors showed up to help us QA / figure out how to approach it, it could probably be done.

Yes, vocab drift matters a lot. Most of the pipeline is basically compressing out any of the weird vocab/language stuff, so it doesn't have numbers and proper nouns and is in a sufficiently abstract concept space to cluster nicely. We use a density-based clustering method (HDBSCAN) which when you set min_samples=1 works as an online clustering (i.e., incrementally adding new things is the same as if you re-ran the clustering in batch mode).

It cost ballpark $250k