Comment by infofarmer

decoding state can be bulkier than a key frame and opaque to the CPU if hardware acceleration is used

I wonder if caching semi-compressed frames would be more efficient in either case (CPU or GPU)