vuciv

Author here. I've always been fascinated by the Sorites Paradox (at what point does a pile of sand become a heap?), so I decided to run an experiment to see how different LLMs handle vague predicates.

I didn't just want a text answer, so I measured the probability logits for "Yes/No" tokens across pile sizes ranging from 1 to 100M grains.

Key takeaways: 1. Prompting "Is this a heap?" directly is useless (the model just agrees with your framing). 2. Few-shot prompting creates a fascinating sigmoid "heapness curve" for most models (Mistral, DeepSeek). 3. Llama-3-8B was the outlier—it remained perpetually uncertain (probs ~0.35-0.55) across almost the entire range. I argue this is actually the most "philosophically honest" reflection of how humans use the word.

I have a feeling that there is an optimal prompt for this type of experiment, but struggle to find it, or even know if I have found it. The charts in the post are rendered in-browser using the data points I collected. Curious to hear your thoughts :)

I would love to work on this. I have worked on this a ton and have really strong ideas for some.

1. I think people assume you have one LLM per character, but I think if you had specialized ones for each quest, item, etc.., this would actually work quite well.

3. I actually think if you cached responses under certain conditions, costs can be saved significantly. This would require quite a robust context, though, to still feel dynamic.

I actually think now that I've gone through the process, memory scanning and writing will be enough... Except, they probably have different control codes that I'd need to reverse engineer.

yes, i'd like to try out Animal Crossing New Horizons next! :)