slewis

Can you try a simpler less programmery version?

"are any of my recent blog drafts unpublished and nearly ready to go?"

It would be really useful to see these evaluated across some of the same evals that the original R1 and deepseek's distills were evaluated on.

Hey, that's me!

Happy to answer any questions about how this works if folks are interested.

I've spent tons of time evaluating o1-preview on SWEBench-Verified.

For one, I speculate OpenAI is using a very basic agent harness to get the results they've published on SWEBench. I believe there is a fair amount of headroom to improve results above what they published, using the same models.

For two, some of the instances, even in SWEBench-Verified, require a bit of "going above and beyond" to get right. One example is an instance where the user states that a TypeError isn't properly handled. The developer who fixed it handled the TypeError but also handled a ValueError, and the golden test checks for both. I don't know how many instances fall in this category, but I suspect its more than on a simpler benchmark like MATH.

Is it stateful? Like can I do a run, read the results, and then do another run from that point?

This 100% matches my experience.

I like to jokingly call founder mode: "fine-grained multi-level oversight". Others might call it the derogatory "micromanagement".

That doesn't mean I control every decision, or that I don't give people space to be creative. What it means is: for whatever is most important for the business, I get involved with the details. The goal is that when I move out of that area, the team I worked with is able to operate closer to founder mode than when I started.

The issue is that vision fundamentally can't be communicated by telephone, or all at once. You're trying to get to a point on the map that most people can't see. The path to it is the integration of all of the tiny decisions everyone makes along the way.

If you only course correct from the highest level you'll never get there.

Overplay one’s hand: spoil one's chance of success through excessive confidence in one's position

The memorization use case is brilliant. Put your talk track for a presentation in and say “help me memorize this by quizzing me”. Thanks!

I call this "keep the fingers moving".

Clicked the youtube link. An hour and 20 minutes later here I am. Thanks for sharing it that was awesome.

I like the ambition!

You can do some amount of iterative and visual data transformation in Weave now. But maybe like 10% of what you can do with Pandas.

Pandas is awesome and lingua franca in data science, unseating it would be an incredible feat if anyone ever pulls that off.

The core data structure in Weave, ArrowWeaveList, has some advantages over a dataframe. It can represent arbitrarily nested data, store references to other object types that can't be stored in arrow (like images), and be visualized in the Weave UI.

For now, we just want to make it really easy to go back and forth.

Hi! I'm Shawn, founder/CTO at Weights & Biases. We've been working on Weave for a couple years now, and it powers core parts of wandb.ai.

It's a UI toolkit built for programmers that can be reprogrammed from the UI itself. You might call it a "yes-code UI" :)

I'm very happy that we were able to release Weave under Apache2 yesterday, and there is a lot of cool new technology in here that we haven't had a chance to describe yet! I'll just try to give a quick tech summary for now.

There are three core components in Weave: Types, Ops, and Panels.

Ops are typed functions like:

  @weave.op()
  def flip_lr(im: Image.Image) -> Image.Image:
      return im.transpose(Image.Transpose.FLIP_LEFT_RIGHT)

Panels are UI elements that register to render a given type:

  @weave.type()
  class BertvizHeadView(weave.Panel):
      id = "BertvizHeadView"
      input_node: weave.Node[huggingface.ModelOutputAttention]

      @weave.op()
      def render(self) -> weave.panels.PanelHtml:
          html = bertviz_head_view(self.input_node)
          return weave.panels.PanelHtml(html)

And they can render other panels.

Panels can expose editable expressions (compositions of Weave ops) to the user, to give them control over what data transforms happen.

We call it Weave because it "Weaves a compute graph through the UI".

Weave includes a vectorizing DAG execution engine built on apache arrow, serialization and data versioning capabilities, and batteries Panels like Table and Plot.

We'll be writing and sharing a lot more now that the release is done. In the meantime, happy to answer any questions!