Comment

Comment by itrummer

The approximation in ThalamusDB is relative to the best accuracy that can be achieved using the associated language models (LLMs). E.g., if ThalamusDB processes a subset of rows using LLMs, it can reason about possible results when applying LLMs to the remaining rows (taking into account all possible outcomes).

In general, when using LLMs, there are no formal guarantees on output quality anymore (but the same applies when using, e.g., human crowd workers for comparable tasks like image classification etc.).

Having said that, we did some experiments evaluating output accuracy for a prior version of ThalamusDB and the results are here: https://dl.acm.org/doi/pdf/10.1145/3654989 We will actually publish more results with the new version within the next few months as well. But, again, no formal guarantees.

parent itrummer 1 year ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton