broner

You can use DuckDB in ipython to solve the globbing issue. Then you don't have to worry about OOMs with geopandas.

I'm doing this with clickhouse querying parquet files on S3 from an EC2 instance in the same region as the S3 bucket (yes DuckDB pretty similar). S3 time to first byte within AWS is 50ms and I get close to saturating an big EC2 instance's 100Gb link doing reads. For OLTP type queries fetching under 1 MB you'll see ~4 round trips + transfer time of compressed data so 150-200 ms latency.

For $1,500 per TB per month? No thanks.

Should be a tax advantage to the non-grind income. $920k with 20% fed + 5% state capital gains tax?

Could you use this to create 100s of on-demand read replicas and get 100s of GB/s out of S3? Seems like a nice way to cheaply and quickly do batch jobs.