sshumaker
Joined 17 years ago
Karma
1,535
Submissions
352
Member since
Oct 16
Recent submissions
Comment · 5 months ago
There’s both medication (which takes a while to get dosage right but can be extremely helpful for some) and also techniques you can use to help compensate.
Comment · 5 months ago
Are you serious? It’s multiple nodes to run a frontier model (a node is 8x GPUs), and they aren’t running on H100s. You are looking at 32+ GPUs.
Comment · 2 years ago
It’s how LLMs work - they are effectively recursive at inference time, after each token is sampled, you feed it back in. You will end up with the same model state (not including noise) as if that had been the original input prompt.
Comment · 2 years ago
It depends on how large the input prompt (previous context) is. Also, if you can keep cache on GPU with a LRU mechanism, for certain workloads it's very efficient.
You can also design an API optimized for batch workloads (say the same core prompt with different data for instruct-style reasoning) - that can result in large savings in those scenarios.
Comment · 2 years ago
They are almost certainly doing this internally for their own chat products.
The simple version of this just involves saving off the KV cache in the attention layers, and restore it back instead of recomputing. It only requires small changes to inference and the attention layers.
The main challenge is being able to do this under scale, e.g. dump the weights out of GPU memory, persist them, and have a system to rapidly reload them as needed (or just regenerate).
Comment · 2 years ago
This is a pretty standard technique if you're running the models yourself. e.g. ChatGPT almost certainly does this.
There's even work that is more sophisticated in this domain that allows 'template' style partial caching: https://arxiv.org/abs/2311.04934
Comment · 2 years ago
Stealth startup | ML Engineer (edge inferencing) | Bay Area or LA (hybrid) | Full-time
We're a stealth startup building something unbelievably ambitious in the AI space that blends AI and gaming tech - co-founders are Andy Gavin [1] (co-founder of video game developer Naughty Dog) and myself (VP @ Microsoft, Credit Karma, previously Google and Naughty Dog). Venture-backed by top investors including First Round Capital and Battery. Get in on the ground floor and work directly alongside a living legend and a dream team of world-class talent.
We're looking for someone who has deep experience with CoreML and optimizing model inferencing for mobile usage (e.g. ANE on iOS).
If interested, reach out to me at: [email protected]
[1] Andy Gavin on the Making of Crash Bandicoot: https://www.youtube.com/watch?v=pSHj5UKSylk
Comment · 2 years ago
I had the pleasure of working closely with Jeff Slutter early in my career. Was the first really fantastic engineer I worked with - he’s at Santa Monica Studios (God of War) these days. I think Matt Toschlog is there too now. Glad to see these folks working on an awesome AAA franchise.
Comment · 2 years ago
Monarchies usually result in long-term thinking? I’m sure we haven’t read the same history books. Most monarchs throughout history have been very self-interested and their decisions have been focused on their own personal interest - the definition of short-term.
I’m not claiming democracy is any better - humans are notoriously bad at long-term thinking unfortunately.
Comment · 2 years ago
Hamas gained a ton of support for their cause by the tragedy inflicted in the counterattack. I believe they celebrate the deaths of Palestinian innocents as much as they do the Israeli ones so they can extract propaganda wins. This has been part and parcel of their strategy for decades - it’s why they embed their military activity in soft targets like schools and hospitals.
The Israelis know this by now, so the fact that Israel was goaded into a ground war speaks as much to the political situation as anything else, but either way it’s tragic.
Comment · 2 years ago
It seems like rather than adding a random amount to each sample (which lets them compute a mean by recreating the same audio and extracting out the differences), Safari could instead add randomness that is based on a key that rotates every hour. (Function of audio sample and key, so the noise would be the same in a given session, but useless for tracking an hour later).
Comment · 2 years ago
Stealth startup | Lead mobile engineer, AI Engineer | Bay Area or LA (hybrid) | Full-time
We're a stealth startup building something unbelievably ambitious in the AI space that blends AI and gaming tech - co-founders are Andy Gavin [1] (co-founder of video game developer Naughty Dog) and myself (VP @ Microsoft, Credit Karma, previously Google and Naughty Dog). Venture-backed by top investors including First Round Capital and Battery. Get in on the ground floor and work directly alongside a living legend and a small team of world-class talent.
Lead Mobile Engineer - Swift and C/C++, CoreML a plus
AI engineer - Finetuning/Retraining LLMs (LLama/Mixtral/etc), MLOps a plus
If interested, reach out to me at: [email protected]
[1] Andy Gavin on the Making of Crash Bandicoot: https://www.youtube.com/watch?v=pSHj5UKSylk
Comment · 2 years ago
You’re being downvoted because this is a hot take that isn’t supported by evidence.
I just tried exactly that with dalle-3 and it worked well.
More to the point, it’s pretty clear LLMs do form a model of the world, that’s exactly how they reason about things. There was some good experiments on this a while back - check out the Othello experiment.
Comment · 2 years ago
You can also look at Bertopic which has this functionality as an open source library:
Comment · 2 years ago
Sometimes you can use a heuristic to estimate K, or use a variant that terminates at some distance threshold.
That said, something like hdbscan doesn’t suffer from this problem.
Product-focused tech executive with 20+ years experience building consumer and enterprise products. Currently co-founder of a stealth startup building something outrageously ambitious.
Previously VP at Microsoft and at Credit Karma. Ex-Googler and former gamedev.