Comment by funcantor

Have you tried WSL? The NVidia developer CUDA repo has a specific folder for "wsl-ubuntu" where you only install the toolkit and it reuses the Windows graphics drivers IIRC.