rastignack

Yes for this particular use case it might help (no per-application aggregated temp file limit though). Also my grid workloads generate thousands of threads running queries with lots of small updates (job queues, calculation results..). I wish there was a way for postgresql to isolate those connections to a limited set of cores (or number of cores).

To avoid completely saturating the server.

Right now we advise using s3 and storing results in files, and asynchronously loading them with batch processes.

Interesting. It would need some work to handle all the workloads I’ve faced where you have two applications with different priorities (ie oltp or grid workloads and analytics). In those cases you want to make sure that BI users do not interrupt the transaction processing by assigning them a set of cores, a priority, and work mem limits for example. Looks doable without major changes.

And why it should be done upstream with a thread pool model and an internal scheduler as MS SQL server does.

The thing is you are going to be comoditized on this too. And it’s going to be faster to migrate off your database with AI. You are going to need to have a plan.

I hope it could be used in the future to export pg_dump formated exports to s3.

One would be able to trigger maintenance jobs via simple lambda functions whose duration is capped.

And if you write to a bytes buffer?

Just monitor it and you’re done. I’ve delivered and maintained hundreds of pg instances and never faced this issue. There is so much literature about it that at some point no one even slightly skilled will face it.

I have tens of clusters to maintain. Quite an advertisement for ECS!

Is there now a way to avoid double buffering and use direct IO in postgresql ?

Has anybody seriously benchmarked this ?

I don’t think io uring would make a difference with this setting but I’m curious, as it’s the default for oracle and sybase.

Remember to turn off autoplay on Twitter.

How about acknowledging you’ve been too sharp with words, apologizing, and attempting to move forward ?

I know other people in the kernel do the same mistake as you frequently do on mailing lists. But two wrongs do not make a right.

Some distributions do release hotfixes before they reach mainline kernel. If you expect end users to compile Linus’ kernel head, why couldn’t they compile your branch though ? They get timely fixes.

Maybe then you should have considered this file system as truly experimental and expected your end users to make frequent backups. And advertise it as such. You could also have some kind of dkms bleeding edge module for your users to test fixes before they reach the kernel.

This way you wouldn’t be so preoccupied about getting code as fast as possible in the kernel.

Maybe the FS was upstreamed too soon, causing your development velocity and the high expectations you have for your end users to be at odd with the well entrenched workflows of Linux maintainers.

In any case as a Linux user, I want to thank you for your work and for your code which taught me a lot.

I hope it didn’t take too much of a toll on you.

Let’s hope that with the recent stabilization, the maintenance will be easier.