flarco
Joined 10 years ago
Joined 10 years ago
Normally you'd keep all the transformation in one place in the "data warehouse", which could be Postgres/Redshift/Snowflake/Clickhouse etc. Using some el tools (like sling cli for example) to load into the DW, and then using a transformation tool (like dbt for example) to do all the transformations in SQL, source controlled. It's advantageous to keep all of the transformation logic in one place (to avoid mismatches as you've mentioned). And from the DW, after data is ready, various tools/clients can consume it (such as BI tools, CRMs, APIs, AI, etc). All aligned.
An alternative tool to Extract/Load data via YAML, Python or CLI.
dbNet is a web-based SQL IDE using Go as a backend, and your browser as front-end. I built it because I was unsatisfied with the database clients out there. Alot of them are top-heavy, unintuitive, slow or expensive.
Here are some of the databases it connects to:
Clickhouse Google BigQuery Google BigTable MySQL Oracle Redshift PostgreSQL SQLite SQL Server Snowflake DuckDB