I think this type of model will have a massive impact on the software industry. 99% of programming tasks in the wild don't involve any kind of algorithmic design, but are more like making a CRUD pattern, writing SQL queries etc. This kind of work is easier to automate but more difficult to source the training data. If and when these models are applied to more mundane problems, I'd expect immediately better performance and utility.
We're also in the very very early days of code generation models. Even I can see some ways to improve AlphaCode:
- the generate->cluster->test process feels like a form of manual feature engineering. This meta layer should be learned as well, possibly with RL
- programming is inherently compositional. Ideally it should perform the generate->cluster->test step for each function and hierarchically build up the whole program, instead of in a single step as it does now
- source code is really meant for humans to read. The canonical form of software is more like the object code produced by the compiler. You can probably just produce this directly
We're also in the very very early days of code generation models. Even I can see some ways to improve AlphaCode:
- the generate->cluster->test process feels like a form of manual feature engineering. This meta layer should be learned as well, possibly with RL
- programming is inherently compositional. Ideally it should perform the generate->cluster->test step for each function and hierarchically build up the whole program, instead of in a single step as it does now
- source code is really meant for humans to read. The canonical form of software is more like the object code produced by the compiler. You can probably just produce this directly