jwuphysics
Joined 15 years ago
Karma
523
Submissions
119
Member since
Apr 5
Recent submissions
Comment · 1 year ago
Have a PhD in physics/astronomy, so—full disclosure—I'm not an expert here... I skimmed the paper and I have no idea what he's talking about.
Given that half of his eight references are to his own papers, and the other half are textbooks or the Heaviside's original work, I think we can assume that he's doing some very niche work or a crank.
Comment · 1 year ago
Could you say a bit more about how so?
Comment · 1 year ago
Incredible, well-documented work -- this is an amazing effort!
Two things that caught my eye were (i) your loss curves and (ii) the assessment of dead latents. Our team also studied SAEs -- trained to reconstruct dense embeddings of paper abstracts rather than individual tokens [1]. We observed a power-law scaling of the lower bound of loss curves, even when we varied the sparsity level and the dimensionality of the SAE latent space. We also were able to totally mitigate dead latents with an auxiliary loss, and we saw smooth sinusoidal patterns throughout training iterations. Not sure if these were due to the specific application we performed (over paper abstracts embeddings) or if they represent more general phenomena.
Comment · 2 years ago
You can get a nice sample of papers using VVV data using the Astrophysics Database System [0]. I mostly study other galaxies, which usually aren't variable on human lifespan-like timescales. Stars can vary on these shorter timescales, and VVV has compiled a huge list of those objects.
At a quick glance, I'd say some interesting results include: * New star clusters discovered in our Galaxy [1] * Galactic maps of dust reddening and stellar metallicity (enriched elemental abundances in stellar photospheres) [2] * Galactic maps of stellar ages throughout the disk plane [3] * Cataloguing other galaxies behind the plane of our own Galaxy [4]
[0] https://ui.adsabs.harvard.edu/search/fq=%7B!type%3Daqp%20v%3... [1] https://ui.adsabs.harvard.edu/#abs/2011A%26A...532A.131B/abs... [2] https://ui.adsabs.harvard.edu/#abs/2011A%26A...534A...3G/abs... [3] https://ui.adsabs.harvard.edu/#abs/2019A%26A...623A.168S/abs... [4] https://ui.adsabs.harvard.edu/#abs/2012AJ....144..127A/abstr...
Comment · 2 years ago
> Why not just say "incoherent output"?
Because the biggest problem with hallucinations is that the output is usually coherent but factually incorrect. I agree that "hallucination" isn't the best word for it... perhaps something like "confabulation" is better.
Comment · 2 years ago
Have you looked into TinyCorp [0]/tinygrad [1], one of the latest endeavors by George Hotz? I've been pretty impressed by the performance. [2]
[0] https://tinygrad.org/ [1] https://github.com/tinygrad/tinygrad [2] https://x.com/realGeorgeHotz/status/1800932122569343043?t=Y6...
Comment · 2 years ago
PBMs hold an incredible amount of power. The Acquired podcast did a marvelous breakdown of the American pharmaceutical industry while covering Novo Nordisk (https://www.acquired.fm/episodes/novo-nordisk-ozempic). If you have three hours to spare, I highly recommend giving it a listen.
Another great interview comes from Mark Cuban, who is serious about disrupting PBMs with his Cost Plus Drug Company (https://www.drugchannels.net/2024/03/mark-cuban-five-ways-th...).
Comment · 2 years ago
That makes sense and I'll run a few evals. Many thanks for open sourcing your work!
Comment · 2 years ago
How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.
Comment · 2 years ago
I am an astronomer and was at a conference when this headline broke. Obviously we ignored this and read the actual papers, e.g. https://arxiv.org/abs/2404.03002. There you can see that the dark energy equation of state parameter w0 = -0.99 +0.15/-0.13, which is not at all significantly different than w0 = -1. That is, no time evolving dark energy. When combined with other cosmological results, the joint analysis slightly favors w0 > -1 rather than w0 = -1 at the ~2.5sigma level or so, but it depends which other results you factor in.
Comment · 2 years ago
Oof, yeah that's not a very good response. But I guess color is actually a technical term in research astronomy: it refers to the difference in two specified magnitudes (usually photometric filters). There's also the question of whether the sun is being viewed from the Earth or from space, since those will change the effective "transmission curve" of the detector system.
I asked: "What color is the sun when viewed from the ground with the human visual system?"
And got the following: "The Sun appears white to us on Earth due to its high temperature and lack of any significant wavelength-dependent absorption or scattering properties. However, if we were able to view it through specialized telescopes that could capture all visible light spectrum (not just the yellow/orange part), then it would appear as an extremely bright ball of blue light with some slight red tint. This is because most of the solar radiation consists of photons at ultraviolet and infrared frequencies which our eyes cannot perceive directly but can be detected by these advanced instruments. [...]"
An overall better response, but still not exactly right. Anyway, the base model was fine-tuned on arXiv/astro-ph abstracts, and I can't imagine too much discussion about the color of the sun in that training data set...
Comment · 2 years ago
UniverseTBD (https://universetbd.org) is also making great strides in the space of large language models and astronomy.
Comment · 2 years ago
Thanks for your spectacular resources! I see that you began an Annotated Mamba repository -- any chance you could share when that blog page might go live?
Comment · 2 years ago
I would be surprised if this is the case. Neural scaling laws are well known and are used by all big industry players to extrapolate experiments.
https://jwuphysics.github.io