This is a tricky problem, but it’s something all good transactional databases solve.
For example, the way this is handled in foundationdb is that you create a transaction, then within that transaction issue a series of read and write calls. The read calls are executed immediately, and they see the database as if other concurrent changes weren’t happening. The write calls get batched up and at the end of the transaction block they are submitted together - with information about which keys were read. On submission, the transaction block can fail with E_RETRY - which means a conflict happened, the transaction was entirely aborted and the userland program should run the transaction again from the start.
I’d absolutely adore an API like this for filesystem operations. There are all sorts of half baked, dirty workarounds to get around the lack of atomicity in posix filesystems. Eg, that whole “write a file then rename it” thing - which almost nobody implements correctly because it’s error prone and most correct solutions on top of posix have terrible performance.
For example, the way this is handled in foundationdb is that you create a transaction, then within that transaction issue a series of read and write calls. The read calls are executed immediately, and they see the database as if other concurrent changes weren’t happening. The write calls get batched up and at the end of the transaction block they are submitted together - with information about which keys were read. On submission, the transaction block can fail with E_RETRY - which means a conflict happened, the transaction was entirely aborted and the userland program should run the transaction again from the start.
I’d absolutely adore an API like this for filesystem operations. There are all sorts of half baked, dirty workarounds to get around the lack of atomicity in posix filesystems. Eg, that whole “write a file then rename it” thing - which almost nobody implements correctly because it’s error prone and most correct solutions on top of posix have terrible performance.