thw20

Joined 2 years ago

Karma

1

Submissions

9

Member since

Apr 24

View on Hacker News

Recent submissions

Simple, zero overhead way to compress model, KV cache via Low-Rank Decomposition jeffreywong20.github.io

1 thw20 5 months ago 0 comments

Comment · 6 months ago

Good work! This is very interesting. Here's a related work that construct low-rank approximation for attention: https://arxiv.org/abs/2505.12942.

Maybe the idea of Query calibration matrix Rxx is of interest to the author!

Comment · 7 months ago

The up to date paper documenting and analysing the observation is now available on arxiv!

Comment · 7 months ago

This project reveals an interesting phenomena, where LLM converts semantic non-informative tokens to attention sinks through middle layer MLP.

The converted sinks are termed secondary attention sinks as they are weaker then BOS attention sinks.

This might be related to layer specialisation in LLM!

Towards understanding multiple attention sinks in LLMs github.com

1 thw20 7 months ago 2 comments

Comment · 8 months ago

This is so amazing. What a masterpiece for intro to reinforcement learning in llm.

The Existence and Behavior of Secondary Attention Sinks arxiv.org

1 thw20 8 months ago 0 comments