woodhull

Our use case requires real-time inserts as events happen.

dbt is as I understand it for batch processing transformations on a set schedule.

We've used PeerDB's hosted offering for sync'ing data from Postgres to Clickhouse both pre and post acquisition by Clickhouse Inc. We've also helped test the integrated sync features in Clickhouse Cloud built on top of PeerDB. We're using it to power customer facing features within our product.

It works well. Their team is great. I feel a bit spoiled having had as much access to the engineering team during the private beta as we've experienced.

It's great for use cases where it makes sense to sync postgres tables across to clickhouse without denormalizing them. PeerDB can transform rows in a single table sent via CDC using a lua scripting language, but it can't (yet!) denormalize data into clickhouse that is stored in 3NF on Postgres across multiple tables.

On the clickhouse query side, we end up wanting denormalized data for query performance and to avoid JOINs. It's frequently not a great idea to query in clickhouse using the same table structure as you're using in your transactional db.

In our experience we sync a few tables with PeerDB but mostly end up using app-level custom code to sync denormalized data into Clickhouse for our core use-cases. Most of the PeerDB sync'd tables end up as Clickhouse Dictionaries which we then use in our queries.

PeerDB works well and I like it for what it is. Just don't expect to be satisfied with querying in Clickhouse against the same table structure as you've got in Postgres unless your data size is tiny.

Curious to know about how others are using it and the architectures you've developed.

We launched our company on Render. It was great for going from zero to one very quickly but we had many problems and ended up migrating to AWS as we scaled.

* Poor visibility into detailed metrics, especially when problems happened in the render load balancer / routing mesh. We had a specific issue where a small number of requests were failing somewhere in render's infrastructure before reaching our application, and at the time there was no visibility to allow customers to know about requests that timed out or failed within render's infrastructure rather than our application. We collaborated with your team to surface and replicate the issue, but it was frustrating. I had a very good set of conversations with a product manager on your team about what we needed and why it was important in early 2024.

* At the time, the hosted postgres implementation was immature. I think this is an area you've already improved dramatically.

* Maybe you could add support for something like AWS PrivateLink so customers can run parts of their workloads on AWS securely over a private network. This would be a neat way to allow customers to stay on Render longer as their needs grow.

For my use case of something similar on Clickhouse:

We load data from postgres tables that are used to build Clickhouse Dictionaries (a hash table for JOIN-ish operations).

The big tables do not arrive via real-time-ish sync from postgres but are bulk-appended using a separate infrastructure.

As much as DuckDB is cute I've mostly come to believe that Clickhouse is the perfect thing to pair Postgres with. This is especially true now that they've acquired PeerDB and are integrating it into the Clickpipes cloud product.

DuckDB is neat, and I understand why a company like BemiDB would build their product on top of it, but as a prospective customer embedded databases are a weird choice for serious workloads when there are other good open-source solutions like Clickhouse available.

What would actually be interesting to me is info on how local data protection authorities across the EU are now interpreting the principles articulated by the ECJ when applied to standard contractual clauses which are what most data transfers actually happen under rather than Privacy Shield.

How all this will play out and be interpreted by regulators is interesting and currently hard to see how everything will be reconciled between trade in digital services, the US national security state, and the fundemental rights guaranteed to EU citizens.

This is from two weeks ago...

what?

Google has been running on 100% renewable electricity since 2017. Microsoft has reached that milestone as well, but I can not find a date for when they achieved it.

AWS aspires to eventually reach that goal by 2025. One of the clouds is dirtier than the others.

Tim in this piece cites the climate pledge as being an admirable Amazon policy.

Unfortunately, it's all lofty language and Amazon lags behind its peers on real action.

For the AWS cloud business Amazon lags far behind its peers at Microsoft and Google. Of the three main public clouds AWS is the only one still using coal power (coal is a big part of the power mix for the grid used by their largest point of presence in Northern Virgina). Microsoft and Google have run their data centers completely on renewables and have done so for years.

We're stuck on us-east-1 in Northern Virginia for legacy reasons and to make up for the dirty way that Amazon runs its cloud we buy feed-in RECs for the grid where our AWS instances run. AWS could be doing this themselves (there are RECs available! we're buying them!) to help jumpstart the transition to renewables in the energy markets where they operate but they're simply choosing not to spend the money.

Microsoft and Google deserve credit for their work in this area and they're doing a much better job. It's just too bad that AWS is a better technical product for our workload.

We're already running separate US/EU points of presence on AWS.

The trouble is that as a small business we can't afford to have two separate operations teams for the US instance vs. the EU instance. We're all based in North America too and it is not practical for us to hire a whole separate devops team for Europe.

Our US based engineers could in theory be compelled to hand over the data stored at rest in the EU. They could also in theory see PII like names or email addresses in the course of administering the application on their laptops in the US which counts as data export, so would still need Privacy Shield or now SCC to allow engineers to do their everyday work keeping the product up and working.

In this particular case all of the people without premium cards who pay with cash or debit cards with no rewards are paying for our free lunch and flight to Hawaii. It's a tax on basically every transaction, paid to those participating in the rewards scheme

The lunch is not free. You're buying it for us.