Modern data warehouses (Snowflake, BigQuery, and maybe Redshift RA3) have incorporated all the key features of data lakes:
- The cost of storage is the same as S3.
- Storage and compute can be scaled independently.
- You can store multiple levels of curation in the same system: a normalized schema that reflects the source, alongside a dimensional schema that has been thoroughly ETL’d.
- Compute can be scaled horizontally to basically any level of parallelism you desire.
Given these facts, it is unclear what rationale still exists for data lakes. The only remaining major advantage of a data lake is that you aren’t subject to as much vendor lock-in.
- The cost of storage is the same as S3.
- Storage and compute can be scaled independently.
- You can store multiple levels of curation in the same system: a normalized schema that reflects the source, alongside a dimensional schema that has been thoroughly ETL’d.
- Compute can be scaled horizontally to basically any level of parallelism you desire.
Given these facts, it is unclear what rationale still exists for data lakes. The only remaining major advantage of a data lake is that you aren’t subject to as much vendor lock-in.