Comment Comment by ai-inquisitor It's not doing that. If you look at the repository, it's adding a new commit with tiny parquet files every 5 minutes. This recent one only was a 20.9 KB parquet file: https://huggingface.co/datasets/open-index/hacker-news/commi... and the ones before it were a median of 5 KB: https://huggingface.co/datasets/open-index/hacker-news/tree/...The bigger concern is how large the git history is going to get on the repository. parent ai-inquisitor 7 months ago
The bigger concern is how large the git history is going to get on the repository.