medvezhenok
Joined 5 years ago
Joined 5 years ago
House occupancy laws also have a role here - some of the living arrangements that ended up in higher density in the past are illegal today, forcing population decline in a given area (which is not necessarily equivalent to _demand_ decline)
I found with using various tricks I can get the zstd parquet to be up to 10x (or more) smaller than an equivalent Postgres table - but obviously the exact compression ratios will depend on the kind of data you have and how well your Postgres table is normalized.
UPDATE my_table
SET x = file1.x,
y = file2.y
FROM 'first_file.csv' file1
LEFT JOIN 's3://my_bucket/second_file.parquet' file2
ON file1.id = file2.id
WHERE mytable.id = file1.id;You can also write a skill that CC can re-use if you're analyzing a lot of similar data sets with minor variance.
It has connectors for Postgres & other stores, so I find it faster to connect to a Postgres instance, pull all of the data from a table (even if the table is like 50GB - if you have 30 cores on the machine it will pull from Postgres using 30 cores in parallel, so it will only take a minute or two) - and then any analytical queries on the data are 10+ times faster in DuckDB over native Postgres (GROUP BY, regexp_replace, count(distinct...) etc).
I do a lot of experiments with regexes, and if you get used to the RE2 syntax that DuckDB uses, you can see up to 10-100x uplift in terms of speed compared to Postgres on things like regexp_matches(), regexp_extract(), etc (depending on query/table/machine specifics). It has quite powerful scripting with custom Macros, fixes a lot of annoyances of SQL for me compared to Postgres.
I think if you have access to a machine with a lot of RAM / cores and a beefy data set, then it's basically like a RAMdisk version of Snowflake running locally on your machine.
(and of course the fact that it makes it convenient to read CSV/parquet, read/write from S3, etc) - it's a very ergonomic tool.
It definitely increases some types of productivity (Opus one-shot a visualization that would have likely taken me at least a day to write before, for work) - although I would have never written this visualization before LLMs (because the effort was not worth it). So I guess it's Jevons Paradox in action somewhat.
In order to observe the productivity increases you need a good scale where the productivity would really matter (the same way that when a benchmark is saturated, like the AIME, it stops telling us anything useful about model improvement)
If you had more like 200GB ram you might be able to run something like MiniMax M2.1 to get last-gen performance at something resembling usable speed - but it's still a far cry from codex on high.
The author uses MIT Living Wage numbers to argue that should be the new "poverty" benchmark - an absurd proposition. Those might be reasonable middle class numbers. He also implies that the benchmark historically represented what is now covered under that $140K calculation - also false; it took ~$9000 in 1966 to cover a "basic standard of living" for a family of 4 with 1 earner; inflation adjusted, that's around $90,000 today. If you add in SS/Medicare taxes (3% then, 15% today), that puts you at ~$100K-105K.
Using the same MIT Living Wage numbers and taking Essex-Princeton NJ as the area (roughly what the author used), you end up with $99,922 as the living wage for a single earner, 4 person household - almost exactly the same as the household back in 1966.
Were there more jobs in 1966 that paid $9000/year versus jobs that pay $100K today? That's the real story you're looking for.
The author uses MIT Living Wage numbers to argue that should be the new "poverty" benchmark - an absurd proposition. Those might be reasonable middle class numbers. He also implies that the benchmark historically represented what is now covered under that $140K calculation - also false; it took ~$9000 in 1966 to cover a "basic standard of living" for a family of 4 with 1 earner; inflation adjusted, that's around $90,000 today. If you add in SS/Medicare taxes (3% then, 15% today), that puts you at ~$100K-105K.
Using the same MIT Living Wage numbers and taking Essex-Princeton NJ as the area (roughly what the author used), you end up with $99,922 as the living wage for a single earner, 4 person household - almost exactly the same as the household back in 1966.
Were there more jobs in 1966 that paid $9000/year versus jobs that pay $100K today? That's the real story you're looking for.