I think this is a clever happy path solution, but it isn't very defensive against problems. If the cloud cron doesn't fire on time or the consumer fails then the queue won't be drained in time. That will cause the events to be batched in the wrong hour.
I don't see how you are building the batches, but there's a lot of complexity around batch sizes, acknowledging that the messages were consumed, etc. Having a single dataset.json.gz per hour feels like a hint this isn't done safely. I'd double check that carefully to avoid losing data.
One tip about compression, I've been really pleased with zstd compression of json/jsonl data.
I don't see how you are building the batches, but there's a lot of complexity around batch sizes, acknowledging that the messages were consumed, etc. Having a single dataset.json.gz per hour feels like a hint this isn't done safely. I'd double check that carefully to avoid losing data.
One tip about compression, I've been really pleased with zstd compression of json/jsonl data.