taylorhou

Running an inference network across 11 Macs (Teale.com), so I pointed my orchestrator at the repo.

The autotuner is the real thing - kernel-level search, config budget split by measured time share, winners cached per device/toolchain. Rotating resident weights across layers to dodge the hot-cache trap is a nice touch.

One question on model ranking: the fit scores look like predictions built from cost constants measured on a single M4 Max, not per-device measurements. How do you rank models across genuinely mixed hardware? Does the estimator improve from actual runs over time? That gap between predicted and measured fit is what eats mixed machine fleets alive.

Shameless plug since magnitude's goal is highly relevant to what i'm working on: teale.com - distributed inference across fleets of macs. If you're running local models on more than one box, check it out with your agent!

yep. as someone who is inbox zero and treats my inbox(es) as task management, it all makes sense to me!

congrats on the public launch! it's been clear to me for a while now that agents will need their own ways to communicate and an asynchronous inbox/task system is a necessity already. will point my agents at the repo to see how we can leverage

is it possible to do this with a wearable? say jewelry? i've been fascinated with the whole digital to physical world dislocation and this inspires me to dust off an old idea of around the world "collecting" passport stamps but the only way to actually collect them is by a physical tap. similarly, i can see jewelry that lights up upon being tapped by a phone or together. fun

i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests.

GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram.

granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.

teale.com - distributed ai inference using networked devices essentially folding@home but sharing underutilized ram (when you're asleep, someone else in the world is awake)

would really appreciate testers but also any companies thinking about distributed inference powered by their own company devices on a private network. my own company has 200+ 16gb ram machines that we're using for inference.

Let's collab. I'm one guy too but I built distributed inference network (teale.com) banging away for about a month with opus/gpt

I built Teale.com and opensourced it. My domain contribution to society. It powers fully distributed inference on Mac, windows, Linux, android, iOS, hell even harmonyOS.

Opensource/weight models will get better and better and eventually we will have mythos level running on smartphone/eyeglass hardware.

It is stupidly tedious currently to match supply with demand though because physical hardware like a 16gb ram MacBook doesn't mean there's truly 16gb available let alone matching models and all of their settings (kvcache, context limit, temperature, etc) to demand.

Would appreciate any help cus we need ai inference by the people for the people.

APMHelp.com / PM-in-a-box.org || AI native growth marketer | Remote | Contract/Full-time/part-time

We're an outsourced accounting firm specializing in property management (appfolio, buildium, rentvine) that's building an open source ai harness/orchestrator for property managers.

$7M ARR, cash flow positive, looking to re-envision marketing/growth ai native.

email taylor at the apmhelp domain or comment and i'll reach out. =)

this is awesome. i'm the founder/maintainer at teale.com (open source distributed inference app) and one of the biggest challenges has been an actually usable/reliable local model that can run on 8gb macbook air's, 6gb android smartphones, etc... will get some of our test machines serving needle

is this available for other open source projects? i'm stealing tokens from my employer effectively and keep hitting my limits re: codex tokens o_O

i'm working on github.com/teale-ai (distributed inference)

250+ employees on the lowest paid subscriptions of claude or openai is $5k/month. their usage of AI is chatbot most of the time which local models/inference can easily handle. so just being able to cancel those subscriptions with local hardware makes my break even on a single $10k mac studio 2 months and honestly the mac studio for their use is overkill.

Distributed ai inference pool for any Mac/iOS device where devices are paid for contributing unused ram. To help with the demand, also doing multiplayer AI. $0.05/million tokens - teale.com