One of the interesting "new direction" for RWKV and Mamba (or any recurrent model), is the monitoring and manipulation of the state in between token. For steerability, alignment, etc =)
Not saying its a good or bad idea, but pointing out that having a fixed state in between has interesting applications in this space
Not saying its a good or bad idea, but pointing out that having a fixed state in between has interesting applications in this space