In such case, imo, it would be more appropriate to use ZIP container format which supports multiple independent entries (files) and index table (including sizes) for them. Compression algorithm is essentially the same (Deflate) so it would not be bloated in any way.
As for practical implementations of serializing complex "objects" into ZIP, numpy [0] can be mentioned as an example.
[0] https://numpy.org/doc/stable/reference/generated/numpy.savez...