Comment by augment_me

Why only loop from the last layer? What if the model can guess that the transformed needed to answer the question better resides in layer 6, and you're at layer 15 out of 128