advaith08

we have access to the thinking trace summaries, not the raw traces

yeah, the tricky thing about the experimental loop is that: 1. its very difficult to do it in a reproducible manner (the same experiment done twice often gives different results due to small undocumented changes) 2. its expensive to do at scale.

Both of these properties make it hard to hill climb on experiment. What's worked for us so far is precisely what you said - having human experts review and provide feedback. we distil their reviews into rubrics, and have LLMs act as proxy experts using these rubrics. We expect the models will hill climb using this approach, and will reach (close to) human expert level by doing this.

thanks for sharing! We think the Experiments-as-code path is not the right approach. the beauty of LLMs is their ability to ingest and reason over unstructured data - they remove the need for formalizing experiments. We tried using declarative templates to document our experiments, but realized that most of the interesting insights (for example, how viscous a liquid feels) is easier described by ranting about the experiment to a LLM, than formalizing it via constructs/code.

interesting! we haven't really considered these yet. However, we may soon have to - the recurring feedback we hear from industry is that crystallinity is a pipe dream, and that most materials are going to be amorphous (or perhaps quasicrystalline). MLIPs have lowered the computation cost for a large number of atoms/odd cell size, but it may be a while before they're accurate enough to simulate these scenarios

that's true, we've seen examples of many promising startups that are a few years in and stuck because the industry is so risk averse. will explore the other domains you've mentioned!

good points. one of the reasons we picked the semiconductor industry is that its less price sensitive than others-companies are willing to pay if the performance is there. Effort is a different story though, and definitely a tradeoff to keep in mind. We're doing experiments ourselves now at university partner labs (UC Berkeley and Stanford), which helps us get moving quickly. At some point, we'll need a partner though - the equipment and testing process quickly get very expensive.

yeah we were surprised by how much it does it. Our approach has been retroactive - we monitor the thinking trace, spot reward hacking behavior and then fix things. We haven't faced this issue with Sol though - its been much more well behaved

We're still figuring this out. We'll need some synthesis equipment (think CVD, PVD etc) and characterization (XRD, Raman spectroscopy) tools in-house to validate that we're making the right materials. We're considering developing these tools in-house - the models sometimes come up with clever modifications to them so that they can deposit new materials. We think equipment is as central to new material discovery as the material itself, and will probably need to be rethought to allow for high-speed AI based experimentation

Cool read, and agree that closing the computation > experimental loop is key!

There’s a variety of computational techniques that help us establish some confidence on the materials. Atomistic simulations can estimate stability and bulk properties of a new material, and we have synthesis experts (min qualification: PhD in thin film deposition) come up with rubrics on how to judge if a material/synthesis recipe is worth trying. All these approaches have known limitations, and improving them is the bulk of our work as a company! There’s also a lot of work to be done in figuring out the minimal set of experiments required to know if a research direction/material set is worth pursuing

The custom instructions to the model say:

"Please note that this is similar but not identical to the antArtifact syntax which is used for Artifacts; sorry for the ambiguity."

They seem to be apologizing to the model in the system prompt?? This is so intriguing