Joined 10 years ago
There may be an economic challenge - which seems to be the sort of problem mass manufacturing can solve very well.
There may be a compute model & latency problem, how do you organize model training when racks are much further apart than in traditional data centres (although speed of light is 50% faster in vacuum than glass fibre). But that's algorithms.
Relative to everything else in orbit, powering a rack of compute and some comms per satellite seems not really to be a physics problem.
This is interesting in that it seems to be making the grind more efficient. I think the true breakthrough will likely be proper scale quantum computing to make the first principles design feasible
For those unfamiliar with the reference, in Hitchikers guide to the galaxy series, Mice are projections of hyper intelligent extra terrestrial beings from a higher dimension who commission the creation of planet earth complete with fossils as a giant organic computer
Even in the top echelon of the Indian government (Indian Administrative Services), a similar culture exists and has interestingly been strengthend as the power has fragmented across political parties. I personally know several of my top B School/Engineering college mates who joined and any of them would do very well in top management in private industry. You can see the result in the rapidity with which India is pulling people out of poverty and modernizing. Certainly alower than China, but pretty amazing seei living outside for a decade now - many urban services are already far superior in quality to what you get in Europe, US or even Singapore. The challenge in India is the bulk of the bureaucracy below them isn't held to the same standard.In India it's still a prestige thing only in the top echelon - the lower echlons are more motivated by job security, pensions and avenues for corruption.
The difference in Singapore is the government pays well enough to attract and retain higher quality talent deeper.
This is nonsense that AI providers want to peddle. Inference is wildly gross margin profitable - likely 90%+ gross margins. It's very easy to work out the cost structures bottoms up. All providers can drop costs to a third and still keep positive gross margins.
The problems are 1. It possibly still doesn't pay out on training investment in a reasonable time frame without a massive expansion of the 90% gross margin.
2. There is no moat. As we see Mac Mini & High End GPUs stock outs and the pricing offered by DeepSeek and Qwen, the performance of Open Weight models are good enough that people can and are already shifting many inference workloads out of these 90% margin players
The first question to ask is does your use case require handling personal or sensitive data.
If you're using the LLM for OpenClaw or you want to handle sensitive or medical data, a local model generally is necessary.
if it's not so sensitive - Cloud providers with some sort of user agreement guarantee on not using your data for training would be the next bet. I personally generally use Gemini or Sonnet as my cloud backup. As I understand, OpenAI, Cloudflare (which bought replicate) and Qwen also seem to provide such guarantees and make SOTA models available. Others like DeepSeek seem to have an opt-out setting. Open router & co I avoid except for benchmarking models with public or dummy data as there is absolutely zero guarantee or ability to enforce terms on providers where your data might be sent.
Gemini and Anthropic (and OpenAI) tend to be expensive - it's very easy to run up 15 dollars a day or so bills which puts you solidly in 1 year pay out on Mac Mini territory - at this point I decided to buy. Gemini Flash Lite 3.1 is however surprisingly good value.
the next question is Mac or CUDA. If your expected use is serving LLM models for inferences, the latest large memory Macs give pretty good inference speed (better than DGX Spark) at a reasonable cost - I think there offer much better value than CUDA if the only use case is LLM inference & harnesses.
if you plan to also fine tune models, experiment with other types of ML on GPUs, do computer vision stuff etc. the development tooling on CUDA is far in advance of all other platforms.
Lastly if you choose CUDA, the question is GB10 family (DGX Spark - cluster able with 128Gb RAM et all) or dedicated GPUs workstations. What I found is practically any serious models weighs in requiring at least 96GB VRAM - Antirez's 2 bit quant of Deepseek 4 flash (my current daily driver) [2] , the Qwen 3.5 122B A10B 4-bit quant, the Qwen 3.6 27B Dense and 35B A3B 8 but quants etc. So you're well out of the consumer GPU territory into 1 or more RTX 6000 Pros or Data center grade devices. Yes you can try to hack away with multiple consumer cards or SSD streaming but it's very fiddly and you probably have better things to do with your life.
The GB10 system - which I ultimately went with - is certainly much cheaper and can be clustered through the Special NVLink cable to get 256, 384 or 512 GB setups but comes with severely constrained bandwidth. The Pro GPUs blast these out of water on performance but are expensive.
Lastly, renting a cloud GPU machine doesn't really make sense except to run already debugged fine tuning workloads. You'll probably spend at least 4 dollar an hour for sufficient capacity which if it's personal use, will mostly sit idle.
1. https://srinathh.medium.com/mid-size-local-models-are-now-co...
