Comment by wenc

For any kind of tabular data, I use DuckDB (parallel CSV reads, works on small to humongous CSVs). It supports full Postgres-like SQL, so you can do arbitrarily complex manipulations on columns and rows. The CSV essentially becomes a dataframe object that can be manipulated in performant ways (DuckDB vectorizes and paralllelizes, so it's way faster than awk on large CSV files). You can even manipulate multiple CSVs in a single statement (JOINs, UNIONs etc).

  % duckdb -c "from 'test.csv'"
  ┌───────┬───────┬───────┐
  │   A   │   B   │   C   │
  │ int64 │ int64 │ int64 │
  ├───────┼───────┼───────┤
  │     1 │     2 │     3 │
  │     4 │     5 │     6 │
  │     7 │     8 │     9 │
  └───────┴───────┴───────┘

  % duckdb -c "select sum(C) from 'test.csv'"
  ┌────────┐
  │ sum(C) │
  │ int128 │
  ├────────┤
  │     18 │
  └────────┘

  % duckdb -c "select sum(A + 2*B + C^2) from 'test.csv'"
  ┌────────────────────────────────┐
  │ sum(((A + (2 * B)) + (C ^ 2))) │
  │             double             │
  ├────────────────────────────────┤
  │                          168.0 │
  └────────────────────────────────┘