cooljoseph

Last year I also had a rather similar idea, but dropped it before I went very far in working on it. I wonder if you and I had similar ideas?

1. Start with an LLM, so that your model understands natural language.

2. Replace RoPE with a tree embedding scheme, and causal attention with a sparse attention on the graph structure. (You could use full attention... but it's cheaper to use graph attention.)

3. Chop off the final unembedding layer, replacing it with a projection down to two scalars, one for logits and one for confidence.

4. Each option of a choice is represented by a number of tokens in leaf position; average these tokens' logit outputs to get the option's logit. Average all of the confidences from all of the options to get the choice's confidence.

5. Train the logits by KL divergence from a true distribution (or NLL on samples from a true distribution).

6. Train the confidences on a subset of the data in which you know the entire true distribution.

The hardest part is getting real world data for workflows, but I wildly speculate that you can get by with only ~50,000 documents if you first adapt domains using synthetic data.

The inputs are natural language, they're just also structured into a tree. The first example on that very page shows natural language instructions:

    questions = {
        "refund_requested": Noul(
            instructions="Does the customer request a refund?",
        ),
    }

A few questions:

1. Do you provide any kind of largest common subtree caching for cheaper input?

2. Have you tried auto-generating Lisp programs structurally?

3. Have you tried augmenting a Lisp language with a `choice` function that makes choices given a prompt, the environment, and the continuation stack?

This sounds somewhat like a normalizing flow from a discrete space to a continuous space. I think there's a way you can rewrite your DDN layer as a normalizing flow which avoids the whole split and prune method.

1. Replace the DDN layer with a flow between images and a latent variable. During training, compute in the direction image -> latent. During inference, compute in the direction latent -> image. 2. For your discrete options 1, ..., k, have trainable latent variables z_1, ..., z_k. This is a "code book".

Training looks like the following: Start with an image and run a flow from the image to the latent space (with conditioning, etc.). Find the closest option z_i, and compute the L2 loss between z_i and your flowed latent variable. Additionally, add a loss corresponding to the log determinant of the Jacobian of the flow. This second loss is the way a normalizing flow avoids mode collapse. Finally, I think you should divide the resulting gradient by the softmax of the negative L2 losses for all the latent variables. This gradient division is done for the same reason as dividing the gradient when training a mixture-of-experts model.

During inference, choose any latent variable z_i and flow from that to a generated image.

> The system is full up at the moment and you can't really add anything without removing some things.

Middle school seems rather un-full to me. Right now students start learning about fractions in 4th grade. They don't move on to algebra until 9th grade. What is there in the middle? Not much, in my experience.

Maybe instead of having a giant no-math gap during middle school, they could move everything down and free up some space later on.

No, surveys like where researchers show up to hospitals and look at the police reports for the injured cyclists.

Sorry if I wasn't clear in my wording. By "survey" I was trying to point to the specific kind of research methodology where you survey people about what has happened in the past instead of trying to control variables like in a typical experiment.

I wasn't talking about random internet polls or self-reported blame analyses, but actual research papers.

> I'd bet accidents are more often than not mostly their fault.

That's actually not true. Most surveys I've seen show that drivers are at fault ~80% of the time.

It is possible to construct φ exactly with a straight-edge and compass. Would the approximation of 5π/6 - 1 be used because it's easier to calculate quickly?

The camera is positioned so that the side window is visible through the front windshield. I think the "black coating" you are seeing is just the interior of the van, and that the entire front windshield is usable.

They seem a lot easier than USAMO problems, or even Putnam problems. I suspect that top students nowadays could easily solve them all in a day.