vlmutolo

They explicitly are only doing lexical matching. I actually don’t mind this if it lowers the cost; LLMs can turn lexical search into a very powerful tool by iterating on the queries. Just look at the success of coding agents using grep and similar code search tools.

But it does seem like their index isn’t great.

What do you think the best use cases for jxl are? Where does it still have an advantage over other formats?

They said 5.6 Sol got something around 40% with the corrected harness.

Another perspective on this is that coordination is the hardest problem to solve in deploying technology that depends on cooperation between different parties. It takes a few of the big centralized players to agree to put their weight behind it.

The tech is actually easier.

It’s a very soft skill that changes as the capabilities of AI change.

But one thing that doesn’t change is the need to specify an end goal correctly and precisely. I think the emphasis of knowledge/information-processing work is going to increasingly be placed on verification mechanisms. This is practically equivalent to precisely defining an end goal.

Spend time deeply thinking about what it means for a solution to be correct. What properties will a correct solution have? What of those properties are testable? Write those things down and tell the agent.

As models get better, agents will be able to target more and more difficult end goals. The strategy just becomes more useful. (It’s useful for people as well.)

For example, if you want an agent to write a photo editor, think about what end properties the editor should have. There are reference images for color space and rendering transformations. That’s a good start.

Sometimes the goal will be fuzzier. “I want a feature like CaptureOne where I provide a reference image and it makes my image look like that.” Well, time to think really hard about what that means. Iterate with AI on how to test for that precisely. Come up with some good metrics/heuristics. Maybe it means local contrast should match. Maybe the overall distribution of colors. Maybe something more complicated.

Then you have a target and you can let an implementation agent work against that. If it fails, it’s either because the agent is bad or your target was incorrect or incomplete. As models get better, the limiting factor becomes your ability to correctly define a problem.

Pretty cool, rendering PowerPoint files to an image is probably the only way for LLMs to make sense of them.

Does this work in Cloudflare’s workerd environment? Would be nice to have a cheap serverless render -> LLM (GLM-OCR / PaddleOCR) -> Markdown pipeline for the various MS Office formats.

If you give every agent an isolated container to use, you’re going to be paying for the reserved memory while the container is active, even if the agent isn’t doing anything.

Wow, that’s very interesting. I wish more benchmarks were reported along with the total cost of running that benchmark. Dollars per token is kind of useless for the reasons you mentioned.

Lots of comments about the price change, but Artifical Analysis reports that 3.1 Flash-Lite (reasoning) used fewer than half of the tokens of 2.5 Flash-Lite (reasoning).

This will likely bring the cost below 2.5 flash-lite for many tasks (depends on the ratio of input to output tokens).

That said, AA also reports that 3.1 FL was 20% more expensive to run for their complete Intelligence index benchmark.

The overall point is that cost is extremely task-dependent, and it doesn’t work to just measure token cost because reasoning can burn so many tokens, reasoning token usage varies by both task and model, and similarly the input/output ratios vary by task.

Very interesting blog post. I’d never seen that method for quickly computing the patterns. I thought I had done a lot of research on bloom filters, too!

Yeah, I agree with this. I think there are open addressing hash tables like Swiss Table that do something similar. IIRC, they have buckets with a portion at the beginning with lossy “fingerprints” of items, which kind of serve a similar purpose as a bloom filter.

This article is a little confusing. I think this is a roundabout way to invent the blocked bloom filter with k=2 bits inserted per element.

It seems like the authors wanted to use a single hash for performance (?). Maybe they correctly determined that naive Bloom filters have poor cache locality and reinvented block bloom filters from there.

Overall, I think block bloom filters should be the default most people reach for. They completely solve the cache locality issues (single cache miss per element lookup), and they sacrifice only like 10–15% space increase to do it. I had a simple implementation running at something like 20ns per query with maybe k=9. It would be about 9x that for native Bloom filters.

There’s some discussion in the article about using a single hash to come up with various indexing locations, but it’s simpler to just think of block bloom filters as:

1. Hash-0 gets you the block index

2. Hash-1 through hash-k get you the bits inside the block

If your implementation slices up a single hash to divide it into multiple smaller hashes, that’s fine.