I've also been struggling to run anything but the smallest model you have shared on paper space:
import torch
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
import torch
from transformers import pipeline
generate_text = pipeline(model="databricks/dolly-v2-6-9b", torch_dtype=torch.bfloat16, trust_remote_code=True, device=0)
generate_text("Explain to me the difference between nuclear fission and fusion.")
Causes the kernel to crash, GPU should be plenty
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P6000 Off | 00000000:00:05.0 Off | Off |
| 26% 45C P8 10W / 250W | 6589MiB / 24576MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
I'm extremely excited to try these models but they are by far the most difficult experience I've ever had trying to do basic inference.
import torch from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
import torch from transformers import pipeline
generate_text = pipeline(model="databricks/dolly-v2-6-9b", torch_dtype=torch.bfloat16, trust_remote_code=True, device=0) generate_text("Explain to me the difference between nuclear fission and fusion.")
Causes the kernel to crash, GPU should be plenty
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P6000 Off | 00000000:00:05.0 Off | Off | | 26% 45C P8 10W / 250W | 6589MiB / 24576MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
I'm extremely excited to try these models but they are by far the most difficult experience I've ever had trying to do basic inference.