kristopolous

boom, jan 2007: https://www.eff.org/deeplinks/2007/01/effs-sweet-16-party-su...

Apparently there's a mitch kapoor connection between the two

https://jeffwerner.ca/kryptonite_lock/

People swapped out their bike locks en masse, it was a pain.

Been around over 20 years

    llcat --save invocation.json \
      -k "@~/credentials.txt:12" \
      -u "@settings.json:.[3].host" \
      -s "@system_prompts:8" \
      -m "@settings.json:.[3].model" \
         "@query.txt:12" > output.txt

   llcat --load invocation.json { ... parameters you want to override ... }

   for i in *; do 
     # use a non-mutable conversation
     # apply it over a group of files with standard unix pipes
     cat "$i" | llcat --load settings.json -cr context.json  > ../results/"${i}.output"

     # or use attachments 
     llcat --load settings.json -a "$i" -cr context.json > ../results/"${i}.output"

     # or save the whole context
     llcat --load settings.json -a "$i" -cr context.json -c ../results/"${i}.output" > /dev/null

   done

   llcat -k @~/secrets.json:.anthropic -u api.anthropic.com 
   llcat -k @~/secrets.json:.openrouter -u openrouter.ai/api

https://github.com/day50-dev/llcat

this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.

Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.

I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest

perplexity has a 27b decision then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.

And you go over to HF and it's even more mysterious https://huggingface.co/models?pipeline_tag=text-classificati...

Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...

Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.

This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.

So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.

I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.

They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted among the providers... and even if you do that 7:2:1 math it doesn't match the providers. Specifically the model in question: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... doesn't match the first listed provider https://deepinfra.com/deepseek-ai/DeepSeek-V4.1-Flash-1017

Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.

Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.

The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.

I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.

It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.

But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.

For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.