rafiki6

No technical analysis, but all models experience drift eventually.

Pretty fun! A few questions:

1 - are you planning to let people write their own prompts?

2 - when will you share the model names?

Because OpenAI is a single team with a singular focus.

Google is a behemoth with multiple products and a lot of people with opinions who you have to get through to launch a product.

Also, OpenAI has not unseated Google's dominance in search nor do I see this happening.

More specifically validation loss is irrelevant when you can't even sample out of distribution anymore.

You seem to have a serious attitude problem in your responses so this is my last one.

It's propietary company evaluation data, and it's for a specific domain related to software development, a domain that OpenAI is actively attempting to improve performance for.

Anyways enjoy your evening. If you want to actually have a reasonable discussion without being unpleasant I'd be happy to discuss further.

Did I say that?

The OP's post was saying it's somehow able to solve something new. It's showing a severe misunderstanding how how language modelling works.

I never said: - "it's not me it's the test" - "These systems are not just next token predictors"

None of the papers or blogs you've shared offer any points that actually rebutt what I'm saying.

And yes, we will eventually have them work in real time. Can't wait.

I have no idea what is shifting in real time. I formed this opinion of GPT4 by running it through several benchmarks and making adjustments to them, so my view is empirical and it was formed 1 week after it came out.

Your post says nothing of substance because it offers no substantial rebuttal and seems to just attack a position by creating a hand-waved argument without any clear understanding of how parameters in-fact impact a model's outputs.

You also completely missed my point.

Just because it's newly created doesn't mean that the structure of the language and the concepts it represents are actually new.

It's clear that whatever tests he writes cover well established and understood concepts.

This is where I believe people are missing the point. GPT4 is not a general intelligence. It is a highly overfit model, but it's overfit to literally every piece of human knowledge.

Language is humanities way of modelling real world concepts. So GPT is able to leverage the relationships we create through our language to real world concepts. It's just learned all language up until today.

It's an incredible knowledge retrieval machine. It can even mimick how our language is used to conduct reasoning very well.

It can't do this efficiently, nor can it actually stumble upon a new insight because it's not being exposed in real time to the real world.

So, this professors 'new' test is not really new. It's just a test that fundamentally has already been modelled.

The problem is everyone is doing this.

There's no moat anymore. Building some dumb webapp to sell to people to make their lives marginally more convenient is not sustainable model.

So what do you do now? Seems like the only option is to move towards a life and death industry.

But when everyone does this, it's game over.

It's easy to say embrace it, but we aren't that far off from a fully capable developer. People are already stitching task based flows to make it basically build a whole thing and a big thing.

The majority of the software industry is in a lot of trouble and honestly given that it was one of the highest paying industries that's not great for the economy.

What are people supposed to even do? What's next for all these displaced workers?

That's pretty much what GPT is.