JoelEinbinder
Joined 5 years ago
Joined 5 years ago
I think your ram usage benchmark is deceptive. I'd expect a minimal browser to have much lower peak memory usage than chrome on a minimal website. But it should even out or get worse as the websites get richer. The nature of web scraping is that the worst sites take up the vast majority of your cpu cycles. I don't think lowering the ram usage of the browser process will have much real world impact.
Getting this page to load quickly was not trivial. The initial dataset of books starting sentences was over 20 megabytes. By only sending the unique prefix of each book, I was able to get that to be much smaller. Using a custom format, sorting the prefixes, and gzipping got the size down to 114kb. About 3 bytes per book. The full first sentences are downloaded on demand as the books are filtered down.
Rendering the books requires 5 million triangles. I used WebGL 2's drawArraysInstanced method. This allows me to define the book geometry only once, and each book is just defined by it's rotation/position/color. Then it's just a matter of keeping the fragment shader simple.
Going into this project, I wasn't sure if it was possible. But I have left feeling really impressed with how capable the web is these days if you are willing to push a bit.
The language models were prompted with the text + each candidate answer, and the one with the lowest perplexity was picked. I tried to avoid instruction tuned models wherever possible to avoid the "voice" problem.