ritz_labringue

Well, it's really a VSCode extension that lets you run Codex CLI in the IDE. Not the "cloud" version of Codex... So GP is technically correct

You’re right, I conflated two things. MoE improves compute efficiency per token (only a few experts run), but it doesn’t meaningfully reduce memory footprint.

For fast inference you typically keep all experts in memory (or shard them), so VRAM still scales with the total number of experts.

Practically, that’s why home setups are wasteful: you buy a GPU for its VRAM capacity, but MoE only activates a fraction of the compute each token, and some experts/devices sit idle (because you are the only one using the model).

MoE does not make batching more efficient, but it demands larger batches to maximize compute utilization and to amortize routing. Dense models batch trivially (same weights every token). MoE batches well once the batch is large enough so each expert has work. So the point isn’t that MoE makes batching better, but that MoE needs bigger batches to reach its best utilization.

The short answer is "batch size". These days, LLMs are what we call "Mixture of Experts", meaning they only activate a small subset of their weights at a time. This makes them a lot more efficient to run at high batch size.

If you try to run GPT4 at home, you'll still need enough VRAM to load the entire model, which means you'll need several H100s (each one costs like $40k). But you will be under-utilizing those cards by a huge amount for personal use.

It's a bit like saying "How come Apple can make iphones for billions of people but I can't even build a single one in my garage"

I don't think I agree with the premise. Sure there are lots of car accidents in absolute terms, but given how many people drive and how error-prone driving inherently is, most people are actually pretty decent drivers

AI is really useful when you already know what code needs to be written. If you can explain it properly, the AI will write it faster than you can and you'll save time because it is quick to check that this is actually the code you wanted to write. So "programming with AI" means programming in your mind and then using the AI to materialize it in the codebase.

Harry Potter action figures trade almost entirely on J. K. Rowling’s expressive choices. Every unlicensed toy competes head‑to‑head with the licensed one and slices off a share of a finite pot of fandom spending. Copyright law treats that as classic market substitution and rightfully lets the author police it.

Dropping the novels into a machine‑learning corpus is a fundamentally different act. The text is not being resold, and the resulting model is not advertised as “official Harry Potter.” The books are just statistical nutrition. One ingredient among millions. Much like a human writer who reads widely before producing new work. No consumer is choosing between “Rowling’s novel” and “the tokens her novel contributed to an LLM,” so there’s no comparable displacement of demand.

In economic terms, the merch market is rivalrous and zero‑sum; the training market is non‑rivalrous and produces no direct substitute good. That asymmetry is why copyright doctrine (and fair‑use case law) treats toy knock‑offs and corpus building very differently.

I very much agree, and I think people who are in denial about the usefulness of these tools are in for a bad time.

I've seen this firsthand multiple times: people who really don't want it to work will (unconsciously or not) sabotage themselves by writing vague prompts or withholding context/tips they'd naturally give a human colleague.

Then when the LLM inevitably fails, they get their "gotcha!" moment.

It does require writing good instructions for the LLM to properly use the tables, and it works best if you carefully pick the tables that your agent is allowed to use beforehand. We have many users that use it for every day work with real data (definitely not toy problems).

It's not on par with o1, let alone o1-pro

Why can't you just create a pool of databases and truncate all tables after each run ? That should be pretty fast

researchers are paid 2x what engineers are paid at OAI, even if it's not the same job there's still one that is "higher level" than the other.

> Create a blockchain-based AI-powered VR platform that allows pet owners to monitor and interact with their pets remotely, providing real-time feedback on their health, behavior, and emotions. Incorporate web3 technology to ensure secure and transparent data management, and leverage VR/AR to create an immersive experience for both pets and owners. This startup idea taps into the growing trend of tech running through every industry, while also addressing the need for innovative solutions that benefit both pets and humans.

All I need now is a tool that will generate the YC application

Thanks ! Didn't know that. Isn't it also very expensive to run ?

Doesn't it cost hundreds of thousands of dollars just to train GPT-3 ? If so, that seems like a good reason to use a "managed" GPT-3.