Coming from a layman's perspective, a genuine question regarding:
"Implements SAE training with auxiliary loss to prevent and revive dead latents, and gradient projection to stabilize training dynamics".
I struggle to understand this phrase "to prevent and revive ", perhaps this is simple speak to those that understand the subject of SAEs, but it feels a bit self contradictory to me, could anyone elaborate?
I struggle to understand this phrase "to prevent and revive ", perhaps this is simple speak to those that understand the subject of SAEs, but it feels a bit self contradictory to me, could anyone elaborate?