jeremytian

this would be an interesting experiment but i think one of the main benefits of having at least some automated pipeline is that human iteration is quite slow since the information intake speed is a few magnitudes slower. probably some combination would be optimal, like ensuring alignment of objectives at the beginning and tuning along the way

From my experience, models that drive optimizers in a loop often get stuck in local minima. ofc, this is just correlation, and not truly indicative, but if you look at the first graph, analyzing precision vs cost, all three model-driven optimizers just bounce back and forth between a precision range, not really expanding the horizon or getting too much worse than the orignal. from our analysis on this data, model-driven optimizers typically don't start from scratch again, they just continue down their optimization path.