0xbadcafebee

> probably pretty good protections

In my career I've seen maybe two companies have pretty good protections. Both had government contracts so they had no choice. Otherwise there's no reason for companies to secure their stack. They make money easier without it.

The buildings are on fire because they're being built out of straw because it's cheaper. Only a government regulation to build out of brick is going to stop the fires.

I don't think they're purposefully bloated, any more than literally all software today is. It all is bloated, but I don't think anyone set out for that. Despite that, the bloat is now the business case. If one of these AI CEOs announced "our main goal is to make tiny models for consumer devices", their stock would crash.

I get that logically you would think that a very wealthy company would improve their technology once they had the resources to do that. But companies do not run by logic. They run by humans, making human decisions, motivated by almost every conceivable thing except logic.

> When you're a startup, expediency and funding prevent this, but by the time you're selling "solutions" and baking designs into ASICs you

.... are still run by the same startup people who think "working like a startup" was a good idea in the first place.

Right now I work with a bunch of people who came up working for startups. They are responsible for the products that make the company the most money. They are still on PHP 7.4.

Everything is enabled by default in Linux, it's just their way. Privilege escalation exploits for Linux are found every year. If you can RCE a Linux box, assume you can also get root.

This is strategic. By making it so consumer grade hardware can't serve it, you have to pay them for access to it, and that funds the server hardware market, which keeps the AI market afloat. If they made smaller models, we would all be running local AI in a few years, nobody would pay them, and they would go bankrupt (and a worldwide recession would begin). It's going to be up to us (FOSS community) to train and publish our own small models. (Also because the US and China are going to make releasing open weights illegal or highly regulated soon)

Open weight is still the correct term because it's not a licensing thing, it's an availability thing. OpenAI/Anthropic are Closed weight because you can't download them at all. Open Source is different because its original use had to do with licensing requirements. Not every phrase with "Open" in it is going to have the same requirements.

When it comes to words, popular usage defines (and redefines) the meaning.

I use this tool every day at work to run different official agent harnesses with subscriptions. The features are plentiful and very useful, upstream development is fast, they accept changes quickly, stable/reliable operation, lightweight. Best of all: it's just a traditional FOSS project, they're not trying to sell you something.

It is created with AI, but I have had no issues using it for months. I'm kind of amazed at how good it is.

It's a model designed to be tiny and still functional, unlike other larger models that are just scaled down and then run into problems (doom loops, intelligence/reasoning failure, tool call failure). You use it when you want a fine-tuned tiny model processing text/audio/video on multiple platforms. (I don't work for them but this is my takeaway)

I love how nobody asks me to give evidence or examples. You don't want to find out you're wrong.

The US is far more hostile than China, also relies on slave labor, and is so increasingly authoritarian that our country's domestic & foreign policy now resembles a 3rd world dictatorship.

Doesn't matter though. The big mistake was allowing manufacturing to be exported from the US to begin with. Nearly any advanced technology we use or wish to produce is now dependent on China. They control not only the advanced manufacturing process, but also the refinement of rare earths (we can mine it but not refine it) and have cheaper labor. Not only can't we make tech without them, we can't sell it at competitive prices. Things will soon get worse as robotics and vertical integration makes Chinese products even cheaper and better quality. And we can't compete on robots because China controls that production chain too.

The smart play would be to build up our own green energy sector to gain cost savings, while investing in its manufacturing, so we can eventually compete and support ourselves (which Biden and Obama passed funding for). Trump eliminated all that when he came back, because the oil companies wanted more profits, they literally don't believe climate change is real, and they have no idea how diplomacy works. We could've had cheap battery, wind, and solar manufacturing in the US. Not anymore.

> and have fun doing it, for better or worse

Like how "the boys" in engineering would made jokes about having sex with the female office manager in the middle of meetings. It was only a great time for certain people.

I kind of forgot about the shared tenancy attack model. I think I was assuming/hoping the software was magically isolating my tokens, when in reality that's impossible for shared tenants. There's whole classes of attacks on things like shared cache, before we get to the LLM-specific features and potential bugs. Enterprises with highly sensitive data should really be on non-shared hosts.