joefourier

How many people are willing to pay more for America/Western-made products? How many are able to correctly know which companies are honest, and which are finding loopholes (e.g. bringing in underpaid temporary workers from abroad, finding the bare minimum for a product to be qualified as "Made in X", writing "Designed in America").

And how many people will be willing to pay a premium when the locally made products are worse quality than foreign products? It's happening right now with Chinese consumer goods (e.g. robotic vacuum cleaners), EVs, etc. Eventually there's a high possibility you won't be able to buy anything in the product category you want.

There's still tons of pure software engineering problems where iterations are time consuming, or there's no obvious solution, like physics simulations, DSP, computer graphics... I can trivially up the difficulty on any tasks that it can easily solve.

LLMs enable me to iterate quicker, accelerate and parallelise, research, scour the literature and integrate cross-disciplinary knowledge, easily implement more complicated algorithms, and are generally better at avoiding trivial mistakes.

Before LLMs, I can't believe how many hours I wasted debugging computer graphics or computer vision algorithms only to realise the problem was my handling of coordinates, flipping x and y, or one program needing the y axis flipped, or typos when copying the algorithm from equation to code. I have no nostalgia for that era.

But why would you throw away that data at all? These are tiny models compared to domain-specific audio/music/image models, and any capacity spent on recognition is capacity that could have been used for prediction.

Is there a reason you don't throw in something like cross-attention on an embedding of the sequence description/data labels? I've trained numerous generative models for audio and images and it's always the case that the unconditional model performs worse, trains slower, etc. than the conditional one (with the best performance from a hybrid using classifier-free guidance or similar scheme).

Again I don't have experience with the domain of time-series forecasting directly, but I would think there are generalities between autoregressive generative models and time-series forecasting, given that both involve predicting the next sample(s) given N regularly sampled past inputs.

My field is primarily audio/vision ML so I don't have any particular domain expertise, but I don't get how you can lump all "time-series" together in one big bucket. My naive understanding is that the way to improve the accuracy is to gain as much joint information as possible. Like:

> Known-future covariates. Some drivers of sensor behaviour are known in advance, such as production schedules, planned shutdowns, weather forecasts. Timestral can condition on them to improve forecast quality, so a scheduled load increase is expected rather than flagged as a surprise.

To me this should be the entire basis of the system, not a condition to improve forecast quality? Like if you're modelling electricity consumption, if all you have is a window of N previous samples, you're going to get completely beaten by a comparatively simple system that knows the time of day, that there's holidays coming up, the weather forecast, etc.

I would also think that these time-series forecasting models would gain from a sophisticated understanding at least the labels of the covariates? A 5M model has no idea what a "bank holiday" is, it just sees an arbitrary class go from 0 to 1, and won't have any useful inductive biases as to what it means or how to link them to other pieces of information.

E.g., wouldn't you predict €/MWh of electricity far more accurately with access to price of natural gas (itself influenced by many geopolitical factors), breakdown in composition of production of each energy source, predictions about roll-out (requiring a sophisticated LLM capable of understanding news and their implications), etc. What's the use in trying to predict it from 2 variables and a narrow window looking back at just 8192 or 32768 samples?

Or perhaps I misunderstand the use-case and this is more for extremely short term movements?

As a mere software engineer, I found that many human written papers in my field (DSP, ML) aren’t exactly well-written, easily understandable or pleasant to read either, and it’s been the case for at least 15 years. In the past, I would frequently spend an hour or two trying to read a SOTA paper full of convoluted prose and dense pages of equations, only to realise that the algorithm is incredibly simple if visualised properly or even just written down as pseudocode.

Now with LLMs the process is much shorter. They can walk me through the algorithm step by step, translate it to pseudocode, explain equations, and most importantly, I can ask them questions and get clarifications, without worrying about seeming ignorant or exhausting their patience. With a paper, you’re stuck with whatever was written (and often, researchers aren’t good teachers, or intentionally over complicate simple concepts).

The list of counter examples is some of the most popular and well-received posts in the rationalist community which are actually in favour of tradition having value.

"The Secret of Our Success" literally starts with how European explorers starved to death in environments that other humans thrived in because they lacked the cultural knowledge local hunter gatherers had, includes many other examples of how pure reason can be counter-productive to survival, and ends with a warning that the idea that now it's finally different may not be true.

I can get past the AI slop design myself, but the writing style is absolutely grating, especially the short repetitive phrases.

"Punchy phrase. No X." "An open engine. Room to make it your own. "Same scene. Same instant. The same explosion. A different atmosphere." "A few things to know. The short answers."

Why is it always the same? What awful post-training, or synthetic data training pipeline, or RLHF gone wrong, produced this homogenised, low variety slop style? And it's totally possible for LLMs to write actually decent or at least varied prose and I would think Anthropic (or OpenAI) has the resources to fix the issue, they just choose not to.

Marketing copy wasn't this homogenous prior to LLMs, unless my memory fails me.

> because you want a specific choice picked or list of things ranked and sometimes the llm will just output something else entirely!

With structured outputs, that just doesn't happen though?

Understandable, (V)AEs are their own rabbit hole (I've spent way too much time working with them). I think they deserve more attention, there's been a few good papers recently about how to ensure that the latent space is optimal for generation without sacrificing too much reconstruction detail, but it's no easy task.

But how do you plan on scaling to 1080p24 (or higher resolution) video with end to end pixel space training? For a 10 second long video, that's 1920x more tokens than your 512x512 DiT experiments, and I'd be concerned about scaling linear patchification temporally and spatially past a certain point.

VAEs don't have to inhibit fine detail, you can trivially increase the channel count like the Flux VAE did, but if you're not careful, that will make training on that latent space much more difficult.

Older models like the SD1.5 work like you mentioned, in large part due to relying on PatchGan which just ensures that individual patches are plausible in isolation. There's nothing preventing a VAE from rendering tiny text with the correct architecture and training. If pushed to an extreme, it could essentially start guessing the letters themselves and write different, but still legible words instead of nonsense scribbles.

I wonder how much of that is due to the use of a suboptimal VAE? The same optimisations could be applied to it, and my intuition tells me that your total compute spend would be even more optimal if you retrained the VAE yourself with a better approach (esp. ensuring translation and rotation invariance, + ability to rescale/blur the latents).

You could also have the same advantages of a draft in low resolution latent space, with an easier to learn data distribution that's more robust to perturbation, but instead of 512x512, you could get a 4096x4096 output for the same compute (assuming a 8x VAE).

There's also the advantage of being able to use a high number of diffusion/flow matching steps for the main LDM while the VAE can be single step and much smaller, since it does not have to handle language or significant scene understanding, just perceptual compression. This sounds especially important for a video model where I would be extremely hesitant to train a generative model without relying on interframe compression.

While I don't know if AI is conscious, I'm sure it has an internal world. Isn't that what a latent space literally is? LLMs operate in a high-dimensional space that can be analysed and translated to very abstract, high level concepts, behavioural tendencies, even things recognisable as emotions. In fact any neural network beyond a single-layer perceptron has a sort of "internal world", it's a matter of complexity and degree.

I don't see how that's true, you can absolutely do the same thing on the desktop? No one is forcing you to use the OS's default widgets there either, you can use Qt, Flutter, or even paint every single pixel yourself which is a lot less feasible on the web.

The main reason the web won is that you can ship SaaS without the user having to install anything locally, and there's no expectation of it being able to work offline, so you can offload processing to servers resulting in fewer bugs, compatibility issues, and DRM problems.

How does that make sense? The expertise is what allows me to tell the LLM what to do in the first place, inspect its outputs, and iterate, which is still faster than coding by hand.