z4y5f3
Joined 3 years ago
Karma
128
Submissions
14
Member since
May 15
Recent submissions
Comment · 2 months ago
Comment · 2 months ago
Yes, and this is not the first time they messed up. They had tokenizer bugs where the trained weights do not match the template back to Qwen3 series.
Comment · 2 months ago
Then "security through secrecy" is really bad mantra especially in the age of AI: others will find the same zero days very soon. If they attack you, then this loses the whole plot. If they propose a fix, then your arsenal becomes smaller.
Comment · 2 months ago
Apparently they are scanning OSS and popular software at scale and disclosing the vulnerabilities they found: https://cvd.z.ai/
Most of these are under embargo, but it seems there are a lot of CVE here from a wide range of popular software, many considered critical or high.
I understand the argument of "people are not actively looking", but isn't the cost for such a scan getting lower by the week, and Anthropic's Project Glasswing is supposed to find them quite a while ago?
Comment · 3 months ago
They will release the weights by 7/27 along with support in vLLM. Stop second guessing. Source: their blog post https://mp.weixin.qq.com/s/V4xhEIy8xDXSMDPrPkmUAQ
Comment · 3 months ago
They will release the full weights by 7/27 along with support in vLLM.
Source: their release blog on WeChat. https://mp.weixin.qq.com/s/V4xhEIy8xDXSMDPrPkmUAQ
Comment · 1 year ago
NVIDIA is likely citing 1 PFlops at FP 4 sparse (they did this for GB200), so that is 128 TFlops BF16 dense, or 2/3 of what RTX 4090 is capable of. I would put the memory bandwidth at 546 GBps, using the same 512 bit LPDDR5X 8533 Mbps as Apple M4 max.
Comment · 2 years ago
Yep I have seen this paper before, and thank you for linking it here for reference. My personal opinion is that compared to single epoch scaling laws, we still need more evidence and literature on effects of multiple epochs, but this paper is one of the best results we have so far on using multiple epochs.
Comment · 2 years ago
What they missed is that current scaling laws (OpenAI, Deepmind Chinchilla) are based on the assumption that the model is trained for one epoch. This essentially means that in order to scale compute, you will have to scale the model size and/or the size of the dataset. So Meta cannot simply spend 3.8e25 FLOPs on a 70B model - to do this they must find 86T pretraining tokens which they do not have.
Of course, ultimately we will figure out scaling laws for LLMs trained on multiple epochs of data, but not today.
Comment · 2 years ago
My experience is that < 500M models are pretty useful when fine-tuned on traditional NLP tasks, such as text classification and sentence/token level labeling. A modern LM with a 32K context window size could be a nice replacement for BERT, RoBERTa, BART.
Comment · 2 years ago
Checked your numbers in another thread - excellent breakdown. Thanks for the clarification.
I did not read the original anandtech post so I did not realize the 512GBps already refers to the full-duplex bandwidth. You are right that PCIe 7.0 x16 sits between V100 and A100 NVLink.
Comment · 2 years ago
NVLink advertises combined bandwidth in both direction, so the 1800 GBps NVLink on Blackwell is actually 900 GBps for everyone else. PCIe can also do multi-node direct transfer via PCIe switches and has been already widely adopted. NVLink still has the power and chip arena advantage even if the bandwidth is similar.
Comment · 2 years ago
Depends. NVLink advertises bidirectional bandwidth, whereas PCIe and standard networking calculate bandwidth in a single direction. So a 1800 GBps NVLink is actually 900 GBps in PCIe and standarding networking terms.
Therefore, a 512 GBps PCIe would sit between the current H100 NVLink (450 GBps) and next generation B200 NVLink (900 GBps). With that being said, NVLink still has lower power draw and smaller chip area, so it would still have a competitive advantage even if the bandwidth is similar.
Comment · 2 years ago
Unfortunately GZIP won't beat LLMs for text classification. The research you cited is just poorly done science that has been widely debunked. The original paper compared top-2 accuracy of GZIP with top-1 accuracy with BERT. The dataset also contains a lot of train/test data leakage. See this article for the rebuttal: https://kenschutte.com/gzip-knn-paper/ and this thread for a previous discussion on hackernews: https://news.ycombinator.com/item?id=36758433.
Further, the evidence presented by NYT in the lawsuit could be hard to reproduce. I tried multiple prompts on multiple versions of GPT-4 APIs but still could not get GPT-4 to reproduce NYT articles exactly. NYT might as well tried to let GPT-4 reproduce 100,000 articles and only found a few cases where GPT-4 actually recited the whole article. In that case OpenAI might as well be arguing that this is only a rare bug and avoid losing the lawsuit in a massive way.
My favorite theory is that they had many model variants internally, each using a slightly different chat template, so when it comes to the release they are not even sure what to use any more.