pickettd

Was the on-device local LLM stack that you tried llama.cpp or something like MLC? I've seen better performance with MLC than llama.cpp in the past - but it has been probably at a least a year since I tested iphones and androids for local inference

I also want to add on that I really appreciate the benchmarks.

When I was working with RAG llama.cpp through RN early last year I had pretty acceptable tok/sec results up through 7-8b quantized models (on phones like the S24+ and iPhone 15pro). MLC was definitely higher tok/sec but it is really tough to beat the community support and availability in the gguf ecosystem.

My gut feeling is that there may be optimization you can do for faster performance (but I could be wrong since I don't know your setup or requirements). In general on a 4090 running between Q6-Q8 quants my tokens/sec have been similar to what I see on cloud providers (for open/local models). The fastest local configuration I've tested is Exllama/TabbyAPI with speculative-decoding (and quantized cache to be able to fit more context)

I experimented with both Exo and llama.cpp in RPC-server mode this week. Using an M3 Max and an M1 Ultra in Exo specifically I was able to get around 13 tok/s on DeepSeek 2.5 236B (using MLX and a 4 bit quant with a very small test prompt - so maybe 140 gigs total of model+cache). It definitely took some trial and error but the Exo community folks were super helpful/responsive with debugging/advice.

As an additional data point - I've read Anathem multiple times and have loved it every time but I read The Book of the New Sun series and really disliked it (though to be fair, I only read TBotNS series once). But I know that TBotNS gets great reviews. So as always with books, your mileage may vary.

The CapOne 2% business card is called Spark Cash and doesn't have the redemption restrictions that you're describing (I think you may be thinking of their Venture card or another personal travel card). The only two downsides to the Spark card (in comparison to the Fidelity cashback card or Citi double cash) are that applying for it likely dings you in all 3 major credit reports (as opposed to just one like most cards) and also that it carries a $59 annual fee after the 1st year.

I got a LG G3 (3GB RAM model) earlier this year and it meets those requirements. I've really enjoyed using it so far.

Or somewhere with great solar power potential. For example Kihei on Maui - you can power a whole house with central air and still have enough energy to charge up an EV just with rooftop photovoltaics.

There is a "Dragon Box" on Amazon that sells for $270ish and appears to be a lower-powered Android Kodi box. I imagine that a second generation Pi could perform just as well.