negativeonehalf

The "whistleblower"'s admission that he "did not have evidence to support his suspicion of fraud" is pretty damning. He fails to meet a much, much lower bar than direct observation -- he admitted he had no evidence at all to even support his suspicion of fraud.

> the most damning evidence you could possibly have is that you cannot reproduce the results

Seems like the "whistleblower" didn't even have that. From the paper by the AlphaChip authors: "We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline. We still do not know how Markov and his collaborators produced the numbers in their paper."

Unfortunately, there aren't publicly available benchmarks for modern technology node sizes, at least not that I'm aware of. Kahng compared on 45nm and 12nm chips, which are very different from a physical design perspective from the 4nm technology node size used by Dimensity 5G, or the sub-10nm technology node size of TPU. "MLContra" used a >100nm technology node size, which is just crazy.

Even if the AlphaChip authors redid Kahng's study properly, this still wouldn't give us useful information -- what matters is AlphaChip's ability to optimize chips in a real-life, production setting, for modern chips, where millions of dollars are on the line.

There's a lot of... passionate discussion in this thread, but we shouldn't lose sight of the big picture -- Google has used AlphaChip in multiple generations of TPU, their flagship AI accelerator. This is a multi-billion dollar project that is strategically critical for the success of the company. The idea that they're secretly making TPUs worse in order to prop up a research paper is just absurd. Google has even expanded their of AlphaChip use to other chips (e.g. Axion).

Meanwhile, MediaTek built on AlphaChip and is using it widely, and announced that it was used to help design Dimensity 5G (4nm technology node size).

I can understand that, when this open-source method first came out, there were some who were skeptical, but we are way beyond that now -- the evidence is just overwhelming.

I'm going to paste here the quotes from the bottom of the blog post, as it seems like a lot of people have missed them:

“AlphaChip’s groundbreaking AI approach revolutionizes a key phase of chip design. At MediaTek, we’ve been pioneering chip design’s floorplanning and macro placement by extending this technique in combination with the industry’s best practices. This paradigm shift not only enhances design efficiency, but also sets new benchmarks for effectiveness, propelling the industry towards future breakthroughs.” --SR Tsai, Senior Vice President of MediaTek

“AlphaChip has inspired an entirely new line of research on reinforcement learning for chip design, cutting across the design flow from logic synthesis to floor planning, timing optimization and beyond. While the details vary, key ideas in the paper including pretrained agents that help guide online search and graph network based circuit representations continue to influence the field, including my own work on RL for logic synthesis. If not already, this work is poised to be one of the landmark papers in machine learning for hardware design.” --Siddharth Garg, Professor of Electrical and Computer Engineering, NYU

"AlphaChip demonstrates the remarkable transformative potential of Reinforcement Learning (RL) in tackling one of the most complex hardware optimization challenges: chip floorplanning. This research not only extends the application of RL beyond its established success in game-playing scenarios to practical, high-impact industrial challenges, but also establishes a robust baseline environment for benchmarking future advancements at the intersection of AI and full-stack chip design. The work's long-term implications are far-reaching, illustrating how hard engineering tasks can be reframed as new avenues for AI-driven optimization in semiconductor technology." --Vijay Janapa Reddi, John L. Loeb Associate Professor of Engineering and Applied Sciences, Harvard University

“Reinforcement learning has profoundly influenced electronic design automation (EDA), particularly by addressing the challenge of data scarcity in AI-driven methods. Despite obstacles including delayed rewards and limited generalization, research has proven reinforcement learning's capability in complex electronic design automation tasks such as floorplanning. This seminal paper has become a cornerstone in reinforcement learning-electronic design automation research and is frequently cited, including in my own work that received the Best Paper Award at the 2023 ACM Design Automation Conference.” --Professor Sung-Kyu Lim, Georgia Institute of Technology

"There are two major forces that are playing a pivotal role in the modern era: semiconductor chip design and AI. This research charted a new path and demonstrated ideas that enabled the electronic design automation (EDA) community to see the power of AI and reinforcement learning for IC design. It has had a seminal impact in the field of AI for chip design and has been critical in influencing our thinking and efforts around establishing a major research conference like IEEE LLM-Aided Design (LAD) for discussion of such impactful ideas." --Ruchir Puri, Chief Scientist, IBM Research; IBM Fellow

Unfortunately, commercial EDA companies generally have restrictive licensing agreements that prohibit direct public comparison.

Still, the fact that Google uses it for TPU is pretty telling - this is a multi-billion dollar, mission-critical chip design effort, and there's no way they'd make TPU worse just to prop up a research paper. MediaTek's production use is also a good indicator.

[flagged]

In the blog post, they announce MediaTek's widespread usage, the deployment in multiple generations of TPU with increasing performance each generation, Axion, etc.

Chips designed with the help of AlphaChip are in datacenters and Samsung phones, right now. That's pretty neat!

The quick-start guide in the repo that said you don't have to pre-train for the sample test case, meaning that you can validate your setup without pre-training. That does not mean you don't need to pre-train! Again, the paper talks at length about the importance of pre-training.

They optimize using a fast heuristic based on wirelength, congestion, and density, but they evaluate with full P&R. It is definitely interesting that they get good timing without explicitly including it in their reward function!