stocknoob
Joined 6 years ago
Joined 6 years ago
For any young programmers: live within your means, invest the difference, become independent, and work on what you enjoy. It’s the best (work related) gift you can give yourself. Skip the self promotion politics unless you enjoy it.
Believe it or not, there are paradigm shifts in technology every now and then. Not everything is 3D television.
It’s too bad they aren’t a fan of dollar cost averaging into broad index funds, I think their older self would thank them.
We aren’t trying to mindlessly consume the same VRAM as last year and hope costs magically drop. We are noticing that we can get last year’s mid-level performance on this year’s low-end model, leading to cost savings at that perf level. The same thing happens next year, leading to a drop in cost at any given perf level over time.
> For training. Not for inference. GPU prices remained about the same, give or take.
See:
https://epoch.ai/blog/trends-in-gpu-price-performance
We don’t care about the absolute price, is the cost per flop or cost per GB decreasing over time with each new GPU?
—-
If it isn’t clear why inference costs at any given performance level will drop given the points above, unfortunately I can’t help you further.
You aren’t trying to run an old 2023 model as is, you’re trying to match its capabilities. The old models just show what capabilities are possible.
https://a16z.com/llmflation-llm-inference-cost/
Look at the log scale slope, especially the orange MMLU > 83 data points.