cswilliams

We saw it when we ran the pg_dump off a standby instance (or a "replica" to use RDS terminology). Our primary was a multi-az instance. So not exactly what they tested here I guess, but it makes me wonder what changes, if any, they've made to postgres under the hood.

Interesting. At a previous company, when we changed the pg_dump command in a backup script to start using parallel workers (-j flag) we started to rarely see errors that suggested inconsistency when restoring the backups (duplicate key errors and fk constraint errors). At the time, I tried reporting the issue to both AWS and on the Postgres mailing list but never got anywhere since I could not easily reproduce it. We eventually gave up and went back to single threaded dumps. I wonder if this issue is related to that behavior we were seeing.

Several years ago at a company I no longer work at, when we swapped from single-threaded pgdumps (from a replica) to multi-threaded pgdumps (-j flag), we occasionally saw some dumps which were "inconsistent". Not sure if that's the best term for it, but we would occasionally get duplicate foreign key errors on pg_restore or even if pg_restore succeeded, sometimes primary key sequences would be less than the maximum id of our table (so new inserts would fail). This was on standard postgres 13.3 (not aurora) on AWS RDS. Nothing really noteworthy about our setup either that I can remember. I tried reporting it at the time on both the postgres mailing list and with AWS support and unfortunately both pointed fingers at the other and I eventually gave up since it was not easy to reproduce. We unfortunately ended up having to go back to single threaded dumps which never produced any of these "corrupt" dumps but was obviously much slower.

Thanks for answering! Unless I'm misunderstanding, one issue with this method is since you're creating a new EBS volume from a snapshot every time the runner starts, the volume will be cold and there will be additional latency on the first reads from the volume. Seems like you could run into this penalty fairly often if you were constantly spinning up and down runners due to inactivity. Maybe something worth considering for v3 (spot instances would be nice to have too).

At first glance, this looks very cool!

On my queue at work coming up is to try to speed up our github action CI runs and I'll definitely take a look at this. Our runs aren't particularly slow by most standards (about 4 minutes), but I would really love to make them much faster. I'm not sure if 1 minute would be possible, but one can dream ;-). But I figure if I can run our test suite on my macbook air m2 in about a minute, I don't see why it's not possible to get my CI near that without spending a fortune. I feel like so much time is wasted in our GHA workflows by downloading the same container images and dependencies over and over. Anecdotally, I also find the GHA hosted runners to sometimes have huge performance swings, where some runs are 25-50% slower for no apparent reason (although time of day seems to affect it). I'm thinking running on EC2 might help with that too.

I've considered some of the third party hosted runners (e.g. buildjet), but didn't love the idea of trusting them with our code base. On the other hand, I looked at some of the projects for running self-hosted gha runners and they seemed like they could require a decent amount of "babysitting", and I didn't see any that supported persistent disks.

Just out of curiosity, can you explain how the persistent disks work in a little more detail? Does it work something like the following:

1. Create EC2 Instance for Runner #1

2. Create new EBS volume and attach it to Runner #1

3. Runner #1 shuts down due to inactivity and EBS volume is detached.

4. Create EC2 Instance For Runner #1 (or does it just stop/start an existing instance?)

5. Attach existing EBS volume created in step #2

Assuming you had multiple runners, would it check for an unattached EBS volume first before trying to create a new one?

Another question I had, do you manage the AMI that the runner uses? Is it the latest ubuntu like GHA uses?

sounds great, will definitely check it out in the new year!

Awesome, just signed up for your waitlist :)

Just out of curiosity, are there any of these 3rd party github action runner services that support persistent disks or have some kind of very fast local cache that can be shared across runners? The majority of time in my workflows is spent downloading the same docker images and dependencies to the runner over and over. I've found Github's own cache to be fairly slow and lackluster.

Looks cool! Does it work with postgres dbs in AWS RDS?

We use it on a large Rails e-commerce site and it works well. All features work as advertised and it is a drop in replacement for mysql. Their support team is also great and is like having your own dba team.

I can also say that I've been participating in the private beta of this AWS version for a week now and it is both stable and performant. Although we use their appliance in production, I'm very excited about this AWS version as it allows us to run clustrix in our testing and staging environments (where we previously had to run mysql).

I think it's a really great thing that more people/companies will be able to use their awesome database without having to purchase an appliance.