JoshCole

> Can you talk more about why you chose CLJ for datascience / ML.

I use Python for a lot of machine learning. My vision transformers, for example, are in Python. There is a lot to like about the Python ecosystem. Throwing away libraries like ablumentations and pytorch because you move to a different ecosystem is a real loss. You probably ought to be using Python if you're doing machine learning of the sort that one immediately thinks of when they see ML.

That said, data science and machine learning are words that cover a lot of ground.

Python often works because it serves as glue code to more optimized libraries. Sometimes, it is annoying to use it as glue code. For example, when you're working on computational game theory problems, the underlying data model tends to be a tree structure and the exploration algorithm explores that tree structure. There is a lot of branching. Vanilla python in such a case is horrifically slow.

I was looking at progress bars in tqdm reporting 10,000 years until the computation was done. I had already reached for numba and done some optimizations. Computational game theory is quite brutal. You're very often reminded that there are less atoms in the universe than objects of interest to correctly calculating what you want to calculate.

Most people use C, C++, and CUDA kernels for the sort of program I was writing. Some people have tried to do things in Python.

> Are there any benefits of using it over Python?

There is an open source implementation of a thing I built. It solves the same problem I solved, but in Python and worse than I solved it and with a lot of missing features. It has a comment in it, discussing that the universe will end before the code would finish, were it to be used at the non-trivial size. The code I wrote worked at the non-trivial size. Clojure, for me, finished. The universe hasn't ended yet. So I can't yet tell you how much faster my code was than the Python code I'm talking about.

> And how is the interop with Python libs?

Worked for me without issue, but I eventually got annoyed that I had to wait for two rounds of dependency resolution in some builds. Conda builds can sometimes have issues with dependency resolution taking an unreasonable amount of time. I was hitting that despite using very few libraries.

I know others already pointed out a ton of things, but having worked with Clojure in 2016 and doing active Clojure development for my startup now I feel like I have to chime in too.

In 2016, Clojure was not great for serious data science. That has changed substantially and not just via Java Interop.

- It now has cross ecosystem GPU support via blueberry libraries like neanderthal, which in benchmarking, outperform some serious Java libraries in this space.

- It has columnar indexed JIT optimized data science libraries via cnuernber and techascent part of the Clojure ecosystem. In benchmarking they've outperformed libraries like numpy.

- The ecosystem around data science is also better. The projects aren't siloed like they used to be. The ecosystem is making things interoperate.

- You can now use Python from Clojure via the lib-pythonclj bindings. In general, CFFI is a lot better, not just for Python.

- The linters are way better than they used to be. The REPL support too.

Clojure already had one of the best efficiency scores in terms of code written to what is accomplished, but now you also get REPL integration, and LLMs have been increasingly capable of leveraging that. There are things like yogthos mycelium experiments to take advantage of that with RLLM calls. So its innovating in interesting new ways too, like cutting bugs in LLM generated code.

It just doesn't feel true to me that innovation isn't occurring. Clojure really has this import antigravity feel to it; things other languages would have to do a new release for, are just libraries that you can grab and try out (or maybe that's the python)

It is pleasantly sensible since servant leadership in a sense creates an innocence because of the incents; the underlying reality is that wisdom dwells with prudence. Principles of revelation arise from incentive compatibility. Aiming at the apparent target is good hearted folly.

Very cool work Jan!

Have you tried experimenting with ham-fisted? I've found the libraries in the techascent part of the Clojure ecosystem to be very good performance wise. Ditto for neanderthal.

Lisps (like Clojure) treat code as data (lists), so you write: `(if x (y) (z))` instead of Python’s `y() if x else z()`. So the code is more concise, but does less to walk a novice through it.

This gains a huge advantage, which allows even more concision: all code is data, so its easy to transform the code.

In Clojure if you want to add support for unless, a thing like if, but evaluating the opposite way you could do this: `(defmacro unless [p a b] `(if (not ~p) ~a ~b))`. Obviously if you wanted to do the same thing in Python you would in practice do `z() if x else y()`. However, you would do it that way because Python isn't as powerful a language. To actually do the same thing in Python you would need to...

1. Add __future__ support.

2. Update the Python language grammar.

3. Add a new AST type.

4. Add a new pass stage to the compiler.

5. Add a python library to integrate with this so you could use it.

Then you could do something like:

    from __future__ import macros

    defmacro unless(pred, then: block, else_: block = []):
        return q[
            if not u(pred):
                u*(then)
            else:
                u*(else_)
        ]

So in the trivial case its just hundreds of lines harder plus requires massive coordination with other people to accomplish the same feat.

This sort of, wow, it takes hundreds or thousands of lines more to accomplish the same thing outside of Lisp as it does to accomplish it within Lisp shows up quite often; consider something like chaining. People write entire libraries to handle function chaining nicely. `a.b().c().d().map(f).map(g)`. Very pretty. Hundreds of lines to enable it, maybe thousands, because it does not come by default in the language.

But in Lisp? In Clojure? Just change the languages usual rules, threading operator and now chaining is omnipresent: `(->> a b c d e (map f) (map g))`. Same code, no need to write wrapper libraries to enable it.

Chess engines cannot see like a human can. When they think they don't necessarily think using the exact same method that a human uses. Yet train a chess engine for a very long time and it can actually end up understanding chess better than a human can.

I do understand the points you are attempting to make. The reason you're failing to prove your point is not because I am failing to understand the thrust of what you were trying to argue.

Imagine you were talking to someone who was a rocket scientist, and you were talking to them about engines and you had an understanding of engines that was predicated on your experience with cars. You start making claims about the nature of engines and they disagree with you they argue with you and they point out all these ways that you're wrong. Is this person going to be doing this because they're not able to understand your points? Or is it more likely that their experience with engines that are different than the engines that you're used to give them a different perspective that forced them to think of the world in a different way than you do?

You're doing this thing again where you say tons of things that aren't true.

> An LLM only learns through input text.

This is false. There already exist LLM which understand more than just text. Relevant search term: multi-modality.

> It doesn't have a first-person 3D experience of the world.

Again false. It is trivial to create such an experience with multi-modality. Just set up an input device which streams that.

> So it can't execute physical experiments, or even understand them.

Here you get confused again. It doesn't follow, based on perceptual modality, that someone can't do or understand experiments. Hellen Keller can be both blind, but also do an experiment.

Beyond just being confused, you also make another false claim. Current LLMs already have the capacity to run experiments and do so. Search terms: tool usage, ReAct loop, AI agents.

> It can understand the texts about it, but it can't visualize it, because it doesn't have a visual experience.

Again, false!

Multi-modal LLMs currently possess the ability to generate images.

> And ultimately our physical world is governed by physical processes. So at the fundamentals of physical reality, the LLMs lack understanding. And therefore will stay dependent on humans educating and correcting it.

Again false. The same sort of reasoning would claim that Hellen Keller couldn't read a book, but braille exists. The ability to acquire information outside an umwelt is a capability that intelligence enables.

For an example of a reward model that doesn't include "C" explicitly consider a reward model defined to be the count of the one bits in letters in the input. It would define a reward for "C" but "C" doesn't show up explicitly, because the reward had universal reach and "C" was among its members as a result.

> But doesn't reward for "*C" means that "C" is in the training data?

You're running into an issue here due to overloading terms. Training data has three different meanings in this conversation depending on which context you are in.

1. The first is the pre-training context in which we're provided a dataset. My words were appropriate in that context.

2. The second is the reinforcement learning setup context in which we don't provide any dataset, but instead provide a reward model. My words were appropriate in that context.

3. The final context is that during the reinforcement learning algorithms operation one of things it does is generate datasets and then learn from them. Here, its true that there exists a dataset in which "C" is defined.

Recall that the important aspect of this discussion has to do with data provenance. We led off with someone claiming that an analog of "C" wasn't provided in the training data by a human explicitly. That means that I only need to establish that "C" doesn't show up in either of the inputs to a learning algorithm. That is case one and that is case two. It is not case three, because upon entering case three the provenance is no longer from humans.

Therefore, the answer to the question but doesn't the reward model for C mean that C is in the training data has the answer: no, it doesn't, because although it appears in case three, it doesn't appear in case one or case two and those were the two cases which were relevant to the question. That is appears in case three is just the mechanism by which the refutation that it could not appear occurs.

> I am not sure if that is an accurate model, but if you think of it as a vectorspace, sure you can generate a lot of vectors from some set of basevectors, but you can never generate a new basevector from others, since they are linearly independent, so there are a bunch of new vectors you can never generate.

Your model of vectors sounds right to me, but your intuitions about it are a little bit off in places.

In machine learning, we introduce non-linearities during training (for example, through activation functions like ReLU or Sigmoid). This breaks the strict linear structure of the model, enabling it to approximate a much wider range of functions. There's a mathematical proof (known as the Universal Approximation Theorem) that shows how this non-linearity allows neural networks to represent virtually any continuous function, regardless of its complexity.

We're not really talking about datasets when we move into a discussion about this. Its closer to a discussion of inductive biases. Inductive bias refers to the assumptions a model makes about the underlying structure, which guide it toward certain types of solutions. If something doesn't map to the structure the inductive bias assumes, it can be possible for the model to be incapable of learning that function successfully.

The last generation of popular architectures used convolutional networks quite often. These baked in an inductive bias about where data that was related to other data was and so made learning some functions difficult or impossible when those assumptions were violated. The current generation of models tends to be built on transformers. Transformers use an attention mechanism that can determine what data to focus on and as a result they are more capable of avoiding the problems that bad inductive bias can create since they can end up figuring out what they are supposed to be paying attention to.