1. Invoice PDFs. Individually small, but there are a hundred million of them. Deleted after 10 years or when the tenant deletes their account.
2. Reports and exports. Few but potentially big files. If an export logically consists of multiple files, it's stored as zip file. Live 30 days or until the tenant deletes their account.
3. Streaming database exports using AWS Database Migration Service for replication into Snowflake
Every file has an entry in the database tracking its storage location and status.
Grouping them by tenant, (sub)type or time-interval makes sense for these. But "dataset" isn't an applicable concept.
1. Invoice PDFs. Individually small, but there are a hundred million of them. Deleted after 10 years or when the tenant deletes their account.
2. Reports and exports. Few but potentially big files. If an export logically consists of multiple files, it's stored as zip file. Live 30 days or until the tenant deletes their account.
3. Streaming database exports using AWS Database Migration Service for replication into Snowflake
Every file has an entry in the database tracking its storage location and status.
Grouping them by tenant, (sub)type or time-interval makes sense for these. But "dataset" isn't an applicable concept.