russli1993

I won't blame Altman here. Memory companies secretly love this. LLM will always be commercialized after people discovered it if it's not Altman it would be another person. And you cannot genuinely say people do not find LLM useful. You cannot genuinely say people will not use LLMs hence there is no demand for compute capacity. OpenAI and others are simply satisfying the compute demand here. Memory companies have secretly loved for an application to drive up the prices of memory chips. And you can't blame them, these foundry workers, physics Phds working at fab companies have worked much harder than your software web app engineers while earning far less. But what they build is a critical enabler of everything digital and much higher barrier to entry. If hyperscaler don't want to pay for exorbitant high memory prices, they can invest and build their own. But why don't they? This is exactly what you are describing free market economy, pricing is based on scarcity. Maybe its time for the world to actually value hardware and semiconductor for their worth.

like anyone don't have a 401k, if AI crashes, say goodbye to 401k. This is not really constructive to the situation. What is constructive is thinking about how to improve the country's electricity system, which will benefit the people and industry alike. This country charges generally 10 to 40% of taxes yet industry have to buy their own generators for power. You know a lot of capex spending by companies which is fueling the AI bubble debate is to cover job functions that should be provided by government with existing tax dollars. And these politicians have no problem voting for 1+ trillion defense spending a year.

I feel the same. I feel machine learning is 1) you need strong math knowledge but also intuition. 2) You need actual production level data, a lot of the nuances of machine learning, applied scientists' roles are how to build strong models with real world, messy, inconvenient data. You need to apply various techniques and not just memorize a specific formula.

Compared to software coding, you build websites, apps etc. The barrier to entry is much higher.

I am trying to pivot into machine learning myself, so far my understanding is try to find a software engineer role where you deal with data, model infra, data pipeline, etc. Then within the same company, try to do more modeling work and build intuition and resume credentials in this area. Someone correct me if I am wrong.

could push kv-cache to off chip cheaper storage like dram. Inference engine manages decode for a single user to run to completion before decode a different users so kv-cache fetch from dram to sram is limited.

what about tiling? seems better to call into existing kernels, fused megakernels than rewrite kernel in java.

since openAI and Anthropic are public about their anti-distillation mechanism, its either their anti-distillation doesn't work or kimi got here because their own training and RLs. Which one is it?

more to say US strategy to strangle Chinese economy and technology industry through semiconductor embargo will fail.

these labs have access to far better NVIDIA chips than deepseek

last year was google winning AI, this year is meta winning AI. I think what is true is Meta, Google will all find their place in the AI world and continue to make huge amount of money. A consumer will use many different AI products, each with a different use case. Everyone will make enough money to live well.

Huawei is launching a mobile soc built with logic folding. They take a single logic ip, like a cpu, and spread it's components across to dies and use hybrid bonding to combine top to bottom. For example the CPU SRAM on top and logic gates directly below. This reduces critical path, increases fmax, reduces power consumption. They claim this method allow them build faster, more power efficient, more function units with duv process than traditional duv. Will be interesting to benchmark after launch later this year

CSPs making own hardware still needs hardware companies, they reduce the Nvidia tax but still need the likes of TSMC, Broadcom, micron/sk hynix, Marvell, the truth semi-companies. CSPs will not have the patents, IPs and talent to replace any of them.

Also, not sure how well CSPs inference stack is compared with vllm + nvidia. A lot of open weight models uses MoE, making the inference stack more complex.

China does not want to crash US economy. US economy strong means more purchases for our goods. But US want to crash Chinese economy by: 2018 trade war, 2019 Huawei sanctions, sanctioning at least 1000 large influential Chinese companies, 2020 Tiktok theft, 2022 total ban on GPUs, semiconductor equipment going to China regardless of end use, banning all sorts of Chinese goods such as EVs, cars, batteries, drones, even DJI cameras, 2020 fake lie about Xinjiang to implement full ban of anything made by Xinjiang, 2025 50%+ tariff on Chinese goods. I don't think building LLM was thought has important until Anthropic went on full China hating, banning anyone on a Chinese IP using the service, even companies headquartered in China's oversea offices. Did you know that a regular person in China trying to use Claude to ask some innocent question gets banned? Talk about everyone created equal and democratic world.

You leave us with no choice but to build technologies ourselves. If its bad for Anthropic, well, you could owned the market in China, oh well, you reap what you sow.

You need 64 H200 super-node for inference for kimi k3. You will not do inference locally. What might pop the western hardware bubble is Chinese GPU, memory, networking companies. But even in China, these AI centric hardware is not cheap.

No, Nvidia is worried about Chinese hardware stack. China is under sanction, what are they training and inferencing these models on? Even if this model might still be Nvidia chips, what about the next model. Everyone knows model and hardware companies are working together. There are a number of very competitive companies in China in this space. After they got this area sorted out, China will do training, inference and tokens entirely on their stack and export their entire stack.

Winners are hardware companies, GPUs,XPUs, HBM, memory, connectivity. Even CSPs are just compute renters, they charge a margin to make sure their hardware purchases can be made back. But given there are more and more AI CSPs, traditional, and neocloud, pricing competition is inevitable, and given the huge expense of hardware, CSPs are squeeze by the hardware companies and users seeking to lower their own costs.