Not to distract from this project — it's very cool! — but I'd love to know if there's something similar in principle to this, but that's focused less on collaborative
querying, and more on collaborative database
administration.
Specifically, I've been in the market for a while now for a webapp you can run on a server, that presents a UX for submitting, watching the progress, and managing, long-running schema changes / data migrations (think: index creations, table re-partitionings) that have been written out as long screeds of custom SQL, and then either submitted through said UI, or which are uploaded to the server out-of-band and then pointed at. (And also, support for kicking off + observing + managing native DB-server commands like Postgres's `vacuumdb --analyze-only` or `pg_dump` / `pg_restore`, and relating them to their DB activity.)
Ideally, the UX would be designed so that you could have many of these jobs running in parallel, without everything descending into incoherence. (I'm picturing: the UI of a torrent client. A table of progress bars + error counts; click in to see the SQL execution log and per-job metrics.)
Ideally also, there'd be a library of SQL templates you could create, where you could generate and fire off N parallel copies of a query with the template-variables populated from some combination of running other SQL queries, and supplied static lists of values; with the meta-job having a configurable max concurrency limit for how many concrete jobs it will ask the DB to schedule at once, separate from the max concurrency of jobs in the DB generally.
Maybe as well, some macro-language for those templates, to let you write "if table_exists(foo), do: [sql], else: [other sql]", or "create_index_if_none_satisfy(table, {colA, colB})", etc — rather than having to look up the unintuitive tables and columns in ANSI-SQL INFORMATION_SCHEMA for the Nth time to then write either "compile-time" SQL or sproc SQL to get this effect.
Right now at $JOB we're doing this kind of thing ad-hoc, by writing scripts that take arguments and query the DB to generate directories full of .sql files, that we then run concurrently using shuf(1), split(1), bash for loops, and named shared tmux(1) sessions; and then visualizing progress in various weak ways, e.g. querying `pg_stat_progress_create_index` during index creation.
Specifically, I've been in the market for a while now for a webapp you can run on a server, that presents a UX for submitting, watching the progress, and managing, long-running schema changes / data migrations (think: index creations, table re-partitionings) that have been written out as long screeds of custom SQL, and then either submitted through said UI, or which are uploaded to the server out-of-band and then pointed at. (And also, support for kicking off + observing + managing native DB-server commands like Postgres's `vacuumdb --analyze-only` or `pg_dump` / `pg_restore`, and relating them to their DB activity.)
Ideally, the UX would be designed so that you could have many of these jobs running in parallel, without everything descending into incoherence. (I'm picturing: the UI of a torrent client. A table of progress bars + error counts; click in to see the SQL execution log and per-job metrics.)
Ideally also, there'd be a library of SQL templates you could create, where you could generate and fire off N parallel copies of a query with the template-variables populated from some combination of running other SQL queries, and supplied static lists of values; with the meta-job having a configurable max concurrency limit for how many concrete jobs it will ask the DB to schedule at once, separate from the max concurrency of jobs in the DB generally.
Maybe as well, some macro-language for those templates, to let you write "if table_exists(foo), do: [sql], else: [other sql]", or "create_index_if_none_satisfy(table, {colA, colB})", etc — rather than having to look up the unintuitive tables and columns in ANSI-SQL INFORMATION_SCHEMA for the Nth time to then write either "compile-time" SQL or sproc SQL to get this effect.
Right now at $JOB we're doing this kind of thing ad-hoc, by writing scripts that take arguments and query the DB to generate directories full of .sql files, that we then run concurrently using shuf(1), split(1), bash for loops, and named shared tmux(1) sessions; and then visualizing progress in various weak ways, e.g. querying `pg_stat_progress_create_index` during index creation.