joelburget
Joined 16 years ago
Karma
694
Submissions
78
Member since
Sep 8
Recent submissions
Comment · 1 year ago
I wrote an OCaml implementation of this paper a few years ago, which I've now extracted into its own [repo](https://github.com/joelburget/constructive-reals/blob/main/C...)
The link in the paper to their Java implementation is now broken: does anyone have a current link?
Comment · 1 year ago
And more recently, [Language Models Use Trigonometry to Do Addition](https://arxiv.org/abs/2502.00873)
Comment · 1 year ago
This is a good idea, though one problem is that Einsum notation (as realized in Numpy and Pytorch) doesn't support the notion of co-contravariance, and the site is based on their Einsum notation. I could potentially add the variances for the examples, though that would move away from how the site currently works (where the information about the reduction comes only from the einsum input).
Comment · 1 year ago
A couple of these I'd like references on if anyone happens to have them.
1. "current science suggests that the actual health impact from consuming most types of plastic might well be essentially zero"
2. "the (weak) evidence we have now suggests running strengthens your knees"
Comment · 2 years ago
o1 is an application of the Bitter Less. To quote Sutton: "The two methods that seem to scale arbitrarily in this way are search and learning." (emphasis mine -- in the original Sutton also emphasized learning).
OpenAI and others have previously pushed the learning side, while neglecting search. Now that gains from adding compute at training time have started to level off, they're adding compute at inference time.
Comment · 2 years ago
Vision Transformers do a shocking amount of compression in the tokenizer. In the [Chameleon paper](https://arxiv.org/pdf/2405.09818) they say the tokenizer "encodes a 512 × 512 image into 1024 discrete tokens from a codebook of size 8192". That's 256 pixels per token (512 * 512 / 1024). If we assume that a pixel is 24 bits (3x 8 bit channels), this implies that they've compressed 256 * 24 = 6144 bits into 13 = (log2(8192)). [An Image is Worth 32 Tokens for Reconstruction and Generation](https://yucornetto.github.io/projects/titok.html) pushes this even further. If these models work similarly, it's no wonder they struggle with some vision tasks.
https://joelburget.com
https://twitter.com/joel_burget
https://github.com/joelburget