> Challenges in long-term planning and task decomposition: Planning over a lengthy history and effectively exploring the solution space remain challenging. LLMs struggle to adjust plans when faced with unexpected errors, making them less robust compared to humans who learn from trial and error.
While working on smol-developer I also eventually landed on the importance of planning as mentioned in my Agents writeup https://www.latent.space/p/agents . I feel some hesitation with suggesting this because it suggests I'm not deep-learning-pilled, but I really wonder how far next-token-prediction can go with planning. When I think about planning I think about mapping out a possibility space, identifying trees of dependencies, assigning priorities, and then solving for some kind of weighted shortest path. That's an awful lot of work to expect of a next token predictor (goodness knows its scaled far beyond what anyone thought - is there any limit to next token prediction?).
If there were one focus area for GPT-5, my money would be on a better architecture capable of planning.
While working on smol-developer I also eventually landed on the importance of planning as mentioned in my Agents writeup https://www.latent.space/p/agents . I feel some hesitation with suggesting this because it suggests I'm not deep-learning-pilled, but I really wonder how far next-token-prediction can go with planning. When I think about planning I think about mapping out a possibility space, identifying trees of dependencies, assigning priorities, and then solving for some kind of weighted shortest path. That's an awful lot of work to expect of a next token predictor (goodness knows its scaled far beyond what anyone thought - is there any limit to next token prediction?).
If there were one focus area for GPT-5, my money would be on a better architecture capable of planning.