Comment

Comment by vivekburman

Not quite exactly,

dbt - code is done via VSCode and managed via git, Job Orchestration is done via Airflow or Dagster

Fivetran - its more of an ELT cloud hosted solution only, and not near real time cases.

Talend, Alteryx - Its a drag and drop first solution. Then comes IDE

My personal work experience - In my current job I was tasked to build dashboard platform which supports near real time sync. I've Postgres as main DB, Elastic search for dashboarding. Due to this near-real time requirement I had to choose a very niche library `pgsync`. Then immediately I rewrote the library again why because pgsync doesn't detect granular change and is tied to redis. Whereas I had rabbitmq as the middle layer for CDC.

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton