exAspArk
Joined 10 years ago
Karma
155
Submissions
76
Member since
Sep 14
Recent submissions
Comment · 1 year ago
Comment · 1 year ago
Hey HN! We’re Evgeny and Arjun. We’ve built a managed version of BemiDB that syncs with your existing PostgreSQL database and gives you fast analytical queries without heavy ETL pipelines.
BemiDB Cloud automatically replicates your data into an Apache Iceberg table stored in your own S3-compatible bucket. The data is compressed columnar Parquet under the hood. We embed DuckDB for query execution, and we speak the Postgres wire protocol so you can use all your existing ORMs, BI tools, or notebooks.
We’ve seen teams push Postgres read replicas too far for analytics or wrangle big ETL flows. We’re aiming for something simpler. Instead of standing up a data warehouse with complex pipelines, you can point BemiDB at your Postgres instance, click to sync, and start querying. All data remains in an open format so you aren’t locked in.
We’d love your feedback! Check us out at https://bemidb.com and our open source repo at https://github.com/BemiHQ/BemiDB. What do you think?
Comment · 2 years ago
Our plan is to make BemiDB work with dbt by leveraging the Postgres-compatibility (supported dbt adapters https://docs.getdbt.com/docs/trusted-adapters). So it should be possible to transform data from Postgres or directly from BemiDB, which may actually perform better.
You're right, the data engineering world is complex, constantly evolving, and has many various solutions. I'd also like to know about any good resources that people use :)
For us, we mostly talked to many potential users asking about their data setups and challenges, and had many conversations with friends and experts in this field. I also read a few weekly newsletters, substracks, and follow people in this space on X (many recently started posting on Bluesky). For a deeper research, reading docs and specs, experimenting, watching talks, listening to podcasts, reading subreddits, etc.
Comment · 2 years ago
This is a great DIY setup. We're hoping to compress this stack and simplify it down to a single binary
1) Read replicas with copied data. The most straightforward, allowing using the same SQL syntax and tooling. Examples: Postgres read replica and BemiDB (disclaimer: I'm a contributor)
2) Operational databases with integrations. Designed for sub-second real-time, bring their own extended SQL syntax for things like window functions. Examples: Materialize and RisingWave
3) Analytical databases with syncing. Allow writing and reading directly, optimized for analytical workloads. Examples: ClickHouse and DuckDB
4) Data warehouses with ETL. Great for large volumes of data, traditionally used with ETL batch processing. Examples: Snowflake and Redshift