jonathanlei
Joined 6 years ago
Karma
202
Submissions
42
Member since
Jul 28
jonathan lei @ voltagepark
View on Hacker NewsRecent submissions
Comment · 1 year ago
It's as difficult as a serverless provider to grow as it was for CPUs before GPUs came along.
Many companies overinvest in fully-owned hardware, rather than renting from clouds. Owning hardware means you underwrite unrented inventory costs and prevents you from scaling. H100 pricing is now lower than any self-hosted option, even without factoring the TCO & headcount.
(Disclaimer: I work at a GPU cloud Voltage Park -- with 24k H100s as low as $2.25/hr [0] -- but Fly.io is not the only one I've noticed purchase hardware when renting might have saved some $$$)
Comment · 1 year ago
Absolutely - GPUs are definitely not a very liquid asset. As someone who works at a GPU neocloud provider (Voltage Park), server assets at scale definitely face a huge slippage, you can buy for $1 and get quotes for $1.50 but only be able to sell for $0.60
Comment · 2 years ago
Hello! If you're interested in monetizing those GPUs, I'd be happy to rent them (all 400!) and offer those to customers of the cloud I work at :)
jonathan [at] tensordock.com
Comment · 2 years ago
Congrats on the launch! This is huge, and it's really cool to see a cloud provider moving this direction - auction pricing for customers so that you always know you're getting the best deal on the market, while providing 100% utilization for you :)
I'm curious what some of the numbers mean: e.g. what does 688/1464 GPUs available indicate in the left gray box? What about there being 1040 GPUs in light gray, and 8 in dark gray?
Comment · 2 years ago
Hmm I did include a training workload as the second chart. My test workload was relatively small so I guess if the workload I ran spends a bit less GPU time comparatively to the CPU, given equal CPU for all workloads, would be an equalizing factor.
But even looking at the Lambda Labs benchmarks, I am surprised that the H100 PCIE barely outperforms the A100 SXM, for example. And it is meant to be a replacement for the A100 PCIE. 20% generational improvement yes, but I would have expected more?
Comment · 2 years ago
Oops, just noticed the link isn’t clickable, here you go!
https://tensordock.com/benchmarks
Comment · 2 years ago
Utilization is not going to be 100% --- ever. Discounts are often given, and servers break down, etc.
Comment · 2 years ago
Thanks! :)
We do have a managed container hosting service, but it's built on our own backend that auto-scales nodes for you when average GPU utilization surpasses a certain point, but it's not K8s -- which would have been a pain to configure given the distributed nature of all of our servers.
Comment · 2 years ago
Hi! Jonathan from TensorDock here :)
We have our own supply base (sourced through https://tensordock.com/host), operate some of our own servers, and are not related to any other marketplace :)
We think we have better security & reliability than Vast.ai due to virtualization rather than Dockerization, as well as more strict access controls [1]. Additionally, you can run Windows VMs on us if you want :)
Comment · 2 years ago
Jonathan from TensorDock (https://tensordock.com/) here - we listed two of our A100 and H100 clusters on the site.
The IB equipped on our clusters (can't speak to others) is 8x 400 Gbps. Most customers training foundational models are able to fully utilize that fabric in parallel.
Comment · 3 years ago
Non-paywalled snapshot: https://archive.ph/r6mkJ