I had this problem too! Jellyfin behind a reverse proxy over Wireguard. For intercontinental visitors (high latency), there would be an initial burst of reasonable transfer speed, but within seconds, slow to an unusable crawl. It took a long time to identify the problem as relating to packet congestion.
Try changing Linux's default congestion control (net.ipv4.tcp_congestion_control) on your Jellyfin & reverse proxy servers to 'bbr'. I don't understand the details- there might be negative consequences [1]- there might be better congestion algos- but for me, this completely solved the issue. Before, connections would stall out to <10%, sometimes even 1% line rate. In quiet/optimal network conditions.
Also, Caddy enables HTTP/3 by default. I force it to HTTP/2.
I should probably investigate using later versions of bbr, though.
Try changing Linux's default congestion control (net.ipv4.tcp_congestion_control) on your Jellyfin & reverse proxy servers to 'bbr'. I don't understand the details- there might be negative consequences [1]- there might be better congestion algos- but for me, this completely solved the issue. Before, connections would stall out to <10%, sometimes even 1% line rate. In quiet/optimal network conditions.
Also, Caddy enables HTTP/3 by default. I force it to HTTP/2.
I should probably investigate using later versions of bbr, though.
[1] https://news.ycombinator.com/item?id=37408406