augment_me

Is there any calculation/projection/cost-benefit done on the effects of these measures? I did not renew my Google AI sub because Gemini started to reject transcribing song lyrics due to IP. This seems like a similar thing

I also see Google winning this in the end. They have absolutely crazy behavioral data and can exploit much more effectively. Meta is also up there, but Google has production hardware as well as clouds, etc.

Its either AGI, or a robbery beyond behavioral data and the whole internet that can make the AI labs win. I dont even know what such a robbery would look like, would perhaps be just making the most advanced hacking tool and then holding the rest of the world and banks hostage.

Let's say you are developing a product, choose 1:

1) Have a single person vibe together a computationally inefficient solution in 1 month that runs transformers on some backend.

2) Hire a team to work for 2 years to design, perfect, optimize, plan, revise, and architect a solution.

The risk/reward for 1) is extremely better than 2). Given the turnover for entertainment in this day and age it doesn't make sense to do all this complex stuff.

But you lose the semantics you choose to not care about. BPE is equally equipped to not fuse single chars if they have downstream value. Most modern tokenizers are already trained with the target text corpus in mind.

For example Google's propabilistic Unigram embedding model is trained jointly with the end task, so if you would train it on a DNA set where individual chars matter it probably would not fuse them

So the interesting question is rather how the authors manage to maintain perf on all modalities/tasks with a single corpus or if they are benchmaxxing something.

This might hold up for theoretical intellectual work without any economical value, but if you are researching anything that can be applied to the industry, you will instead have 1 year of market ownership and first-mover advantage.

I think it says more about what the value of math as intellectual work is, than about intellectual work in general.

Like if an Egyptian historian would come and say "let's wait one year until the methods are good enough to decrypt this language", no one would be bothered or surprised.

The newest submissions to Hutter Prize made crazy jumps by just replacing the LSTM with a transformer and training it.

35'000$ for a prompt and some training is a nice chunk of cash.

Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.

Its cool, but it goes against the whole optimization that NLP has gone through with BPE and redundancy/repetition in large data in general.

Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.

Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.

The amount of upselling, extra parts necessary for basic functionality, predatory resin pricing and specific open "modes" that are free to be subjects to amendment by contract makes this seem like a cash grab for investors rather than serious product. Really sour taste in the mouth from this.

Just buy an open source version from China that has the necessary hardware in an open format and let Claude adjust the software if there are concerns.

It's more like it doesn't matter if you sell the most organic, free range, ethically sourced, nutritious, tasty ice cream if your competition is allowed to put cocaine in theirs and get all your customers.

TikTok would survive and do just fine even if Instagram decided to cede the market to them

I am afraid there is a demand for the torment nexus, and until govt steps in, you're out of luck and behavioural surplus is there for the taking by any means.

Humans are bound by communication overhead and non-market-based incentives. Adding 3x engineers does not do anything. On an average workplace, you have 80-90% of dead weight who shuffle pointless information around and play social games to keep their children fed.

Giving the 10% access to these tools is a crazy value multiplier, because they can run at lightning speed with their mental models and the right incentives for the company.

You can take the right path and in the end no-one wants to use your product because it's not as engaging.

Instagram tried to take the "good" path in 2014 and lost 90% market share to TikTok who attentionmaxxed things. Then they had to take the exploitative path to claw back users.

As long as it's not regulated for everyone at the same time, you need to do this for survival of your product.

Why is #2 important to answer?

Was the question of if horses and carriages are important a thing we had to answer? No, it just fell out of it. If human maths is pointless, what is the issue with defunding it?

I think that this was true pre-attention economy. If you look at security incidents and other prod issues, they rarely move the needle today. I work at a hyperscalar and we have been shipping 40% more bugs to prod since 2023 year-to-year, yet user growth and revenue is up about 25% similarly due to more features and more ads.

I think that the whole premise of that failure affects things is based on a older model. You kind of assume that:

1) users are able to easily gain information about alternatives. We have seen that since LLMs replaced search engines, we have much less churn.

2) users are not behaviorally addicted/pushed into patterns. We have patterns that utilize users to keep using the app by exploiting their cognitive weaknesses

3) users have the mental fortitude and stamina to learn new tools instead of accepting flaws in the one they are used to

Some failures and bugs in prod don't matters as much