cheeseblubber

AI intelligence is also very jagged. Meaning it can be super human in a lot of things and bad at a ton of others. This is most represented in places where they can't get enough data. Starting business is a combination of a lot of skills chained together and breaking a single chain can cause the business to not work.

Secondly even they are constrained by resources. They have a limited amount of people/GPUs/capital. Given the constraints its best to focus your attention on the highest ROI given your team and organization.

Anecdotally the site we manage we are easily seeing 100x the traffic from bots than from humans in the past year. So much so that it is impacting our hosting costs.

My problem with the article is that they don't even mention this fact. The metaphors with Uber often is brought up but it breaks down at cost optimization. It also wouldn't be fair to say we are at the peak efficiency of LLMs and that there wouldn't be any improvements left.

It make sense if you account for cost of intelligence getting cheaper every year. Most of the models per unit of intelligence is getting far cheaper. We get better hardware, architecture, training techniques, inference optimizations and caching. All those improvements add up. In in early 2022 you were getting 10x cheaper annually now is closer to 2x - 5x cheaper annually. The cost is still dropping where as Uber can only get the cost down by so much.

The human brain also is a product of billions of years of evolution. We branched off from our common ancestor 7-9 million years ago. We encode quite a lot of structure and information that is essential for intelligence. The starting point of just our life time of training is incomplete.

If you calculate 100W * 7 million years * 365 = 255,500MW to train.

Yes definitely we were using our own budget and out of our own pocket and these model runs were getting expensive. Claude costed us around 200-300 dollars a 8 month run for example. We want to scale it and get more statistically significant results but wanted to share something in the interim.

Via api you can turn off websearch internally. We provided all the models with their own custom tools that only provided data up to the date of the backtest.

We used the LLMs API and provided custom tools like a stock ticker tool that only gave stock price information for that date of backtest for the model. We did this for news apis, technical indicator apis etc. It took quite a long time to make sure that there weren't any data leakage. The whole process took us about a month or two to build out.

You can click in to the chart and see the conversation as well as for each trade what was the reasoning it gave for it