Joined 9 years ago
Postgres 16 added support for replication slots on stand-by servers, which allows you to point Debezium to a promoted stand-by after failover and resume from there, no events will be missed. It required manual advancement of the stand-by slots though [1]. Postgres 17 brought full support for failover slots, fully automating the entire process [2].
[1] https://www.morling.dev/blog/logical-replication-from-postgr... [2] https://www.morling.dev/blog/failover-replication-slots-with...
One big problem I have with them is that they take away time from project maintainers for reviewing and helping to get the PRs into shape, which then can't be spent on other, more important things. I feel like the "good first issue" GitHub label is specifically attracting these kinds of contributions.
It's not a black-or-white thing though, and you need to tell apart folks who produce slop PRs against any arbitrary repo, from folks using AI to contribute in a sensible way. We've tried to codify some rules in our contribution guide [1]:
- PRs from apparent bot accounts are closed - PRs from users who file large numbers against random repos are closed - You're welcome to use AI, but you need to stand behind your PR and be able to explain it
I'm sure we'll adjust those rules over time, but since we have instantiated them, it definitely has become easier to deal with AI PRs and handle them in an a relatively objective way. It absolutely means that sometimes a PR will be closed which could have been an improvement, but I think this is the right thing to do given the circumstances.
[1] https://github.com/hardwood-hq/hardwood/blob/main/CONTRIBUTI...
From the post:
> As an example, the values of three out of 20 columns of the NYC taxi ride data set (a subset of 119 files overall, ~9.2 GB total, ~650M rows) can be summed up in ~2.7 sec using the row reader API with indexed access on my MacBook Pro M3 Max with 16 CPU cores. With the column reader API, the same task takes ~1.2 sec.
In my measurements, this is significantly faster than parquet-java for the same task (which is not surprising, as Hardwood is multi-threaded); but I want to be sure I am setting up and configuring parquet-java correctly before publishing any comparisons. The test above also is hooked up to run parquet-java (and there's a set-up for PyArrow, too), so you could run it yourself on your machine if you wanted to.
So far, we've spent most time optimizing for flat (non-nested) data sets which are fully parsed (either all columns, or with projections) and I think it's faring really well for those. There's no support for predicate push-down yet, so right now, Hardwood isn't optimal for use cases with high query selectivity; this is the next thing on the roadmap though.
Approaching the home stretch for a first 1.0 preview release, including: support for parsing Parquet files with flat and nested schemas, all physical and logical column types, core and advanced encodings, projections, compression, multi-threading, etc. all that with a pretty decent performance.
Next on the roadmap are SIMD support, predicate push-down (bloom filters, statistics, etc.), writer support.
Reading through the README, this piqued my curiosity:
> Small or fast transactions may share the same WAL position.
I don't think that's true; each data change and each commit (whether explicit or not) has its own dedicated LSN.
> LSNs should be treated as monotonic but not dense.
That's not correct; commit LSNs are monotonically increasing, and within a transaction, event LSNs are monotonically increasing. I.e. the tuple commit-LSN/event-LSN is monotonically increasing, but not LSNs per se. You can run multiple concurrent transactions to observe this.
