rythie
Joined 18 years ago
Joined 18 years ago
Cameras also need to withstand drops for similar reasons to phones, it’s in you hand and you could drop it, also tripods can fall over, car mounts fall off etc.
I can have two (or more) batteries, if it runs out I just change it. I don’t need walk around with a USB battery pack and cable hanging off the device preventing me from using it properly.
I can put the battery on charge somewhere and leave it, even if not completely secure, because just the battery not the device. This way my expensive device and my data is not at risk.
I can use 40+ year old cameras, because I can just put a new battery in. This is not something you can do with newer device, e.g. and iPod and you can’t even find anyone who will fit them for the older models.
Battery tech moves on. There are now some batteries with charging ports on them. Other batteries offer more capacity than the original ones. Apple even did this once for me, when MacBook Air batteries were fairly easy to replace, I had mine replaced (it wore out) at the shop and they put a slightly bigger one in, which was the standard on the newer models.
For OpenAI, I’d assume that a GPU is dedicated to your task from the point you press enter to the point it finishes writing. I would think most of the 700 million barely use ChatGPT and a small proportion use it a lot and likely would need to pay due to the limits. Most of the time you have the website/app open I’d think you are either reading what it has written, writing something or it’s just open in the background, so ChatGPT isn’t doing anything in that time. If we assume 20 queries a week taking 25 seconds each. That’s 8.33 minutes a week. That would mean a single GPU could serve up to 1209 users, meaning for 700 million users you’d need at least 578,703 GPUs. Sam Altman has said OpenAI is due to have over a million GPUs by the end of year.
I’ve found that the inference speed on newer GPUs is barely faster than older ones (perhaps it’s memory speed limited?). They could be using older clusters of V100, A100 or even H100 GPUs for inference if they can get the model to fit or multiple GPUs if it doesn’t fit. A100s were available in 40GB and 80GB versions.
I would think they use a queuing system to allocate your message to a GPU. Slurm is widely used in HPC compute clusters, so might use that, though likely they have rolled their own system for inference.
[1] https://history-computer.com/palm-pilot-guide/ [2] https://www.zdnet.com/article/pocket-pc-sales-1-million-and-...
The latest Nvidia driver no longer supports the K40, so you’ll have to use version 470 (or lower, officially Nvidia says 460, but 470 seems to work). That supports CUDA 11.4 natively. Newer versions of CUDA 11.x are supported: https://docs.nvidia.com/deploy/cuda-compatibility/index.html though CUDA 12 is not.
In my testing, a system with a single RTX3060 was faster in tensorflow than with 3 K40s and probably close to the performance of 4 k40s.
If you are considering other GPUs, there are some good benchmarks here (The RTX3060 is not there, though the GTX1080Ti was almost the same performance in the tensorflow test they run): https://lambdalabs.com/gpu-benchmarks
As others have said Google CoLab is free option you can use.