Comment

Comment by akubera

What makes you think it's not compressed? (or that the data is stored as XML?)

There's very sophisticated compression systems throughout each experiment's data acquisition pipelines. For example this paper[1] describes the ALICE experiment's system for Run3, involving FPGAs and GPUs to be able to handle 3.5TB/s from all the detectors. This one [2] outlines how HL-LHC & CMS use neural networks to fine tune compression algorithms on a per-detector basis.

Not to mention your standard data files are ROOT TFiles with TTrees which store arrays of compressed objects.

It's all pretty neat.

[1] https://arxiv.org/pdf/2106.03636

[2] https://arxiv.org/pdf/2105.01683

parent akubera 2 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton