armcat

First off - well done, this is amazing. Secondly, it's a fascinating read how you basically figured it out, one mistake at a time, like not connecting or fastening the battery correctly. This is the way! We make mistakes and figure things out in our journey to make something interesting, and learn along the way.

This opens up so many interesting opportunities. In principle one could compile a small VM inside the LLM weights. The VM can host for example a Python interpreter, a shell and tools like sed, cat, grep etc. Instead of LLMs having to do slow autoregressive token emission with the current "external" tool calling, it could literally execute everything except web search, inside the LLM, with just matmul operations. You could probably optimise this so it runs in batches. You could have a gdb like application that can step through the LLM forward pass. Anyway, just a crazy thought. Have a great weekend everyone!

The article doesn't establish whether AI ultimately benefits attackers more than defenders. Open weight models give defenders access to the same capabilities. At the very least the community has a level of symmetry they wouldn't otherwise have.

There is a human/user element here as well - will the users check the citations, if they are provided inline? I recall in the early ChatGPT web search and Perplexity days, most of the citations didn't work (made up, or 404). You get a false sense of security by having a "[1]" next to the stat or a claim. Maybe labs companies should have some additional quality ranking/checks on these, if it's in their interest of course.

I know it was tested in some "trial classrooms" and the response was very positive but I've left that company since and don't know the status. It was part of a UN programme as well.

But essentially it's absolutely possible to have a little cigarette box thing with most of the worlds knowledge at your fingertips, completely offline, and it doesnt require LLMs but just traditional information retrieval techniques.

I worked in legal tech/AI. It's relatively easy to reach the capabilities of Legora and Harvey. My finding is that it's the industry itself that's very tricky and old fashioned. Lot of legal contracts at the large law firms (which Legora/Harvey are targeting) are controlled by iManage and NetDocuments systems. To integrate into their workflows you have to be "chosen" as a development partner. The said law firms also care very deeply about security and audit trails, where the documents reside, how they are processed, etc - at least on the surface. Despite their size and wealth they have invested very little in their in-house data & IT capabilities - maybe they have couple of techies working full time, but mostly they outsource to other companies. They also still print documents and do manual redline. And on top of that you have this insane GTM from Legora/Harvey where their sales peeps are crazy good, and work insane hours.

So it's not really a technical challenge, it's more of an industry challenge, so they will happily buy Legora/Harvey licenses. But maybe (probably) this will change over time.

Median of less than 2 seconds for final word after its spoken on esp32 is insane, amazing work!