I don't think you understand the point of the constraints ZFS has put in place here. They're not there for transactional isolation per se. They're there because these ZFS maintenance operations run as
synchronous system calls that take per-filesystem global mutex locks.
Blocking within one, with those locks acquired, and thunking back down to userspace, would mean that 1. a CPU core, and 2. the filesystem itself, would both be tied up
indefinitely until that callback thunk returns. If it ever does!
Picture a database server where once you begin a transaction, the whole DB server goes single-threaded and doesn't serve any other clients until the TX completes. Fundamentally, at least for its more arcane global operations, that's exactly what a filesystem is. Usually this doesn't matter, because even these arcane filesystem operations take on the order of microseconds to complete (just with lots of non-locked pre/post execution overhead that inflates the total syscall time.) But a userspace program can do arbitrary-much stuff in a callback. It can even just get into an infinite loop, and never get back to the kernel. This is why there isn't such a thing as a system call with userspace callbacks! (Or any API equivalent to one—e.g. one with tx_begin + tx_commit calls, where tx_begin acquires a global kernel lock before returning to userspace, under the expectation that userspace call tx_commit to release the kernel lock.)
On the other hand, submitting a "callback program" as a whole to the kernel, allows the kernel to statically analyze the behavior of the "callback program" in advance; and only if it is determined to be "simple" (e.g. if it predictably halts after a short time as a static property), will the kernel issue the program a capability for actually running that "callback program."
This is how compute shaders work. This is how eBPF works. And this is (apparently) how this ZFS Lua thing works. They all do it for similar reasons: to ensure that the program is a complete, soft-realtime practical unit of work to be executing in some bounded-time context.
You do get atomicity from this, but it's not MVCC atomicity. It's more like Redis's atomicity: during the execution of a ZFS Lua program, all other clients making system calls against the filesystem will wait on acquiring the filesystem mutex, and so will block until the program completes. There's never a case where other activity might "interleave" with the Lua program's execution, and cause it to retry. There is no other activity. The only case in which the Lua program can actually fail (and so get rolled back), is if it dies/is killed in the middle of execution, due to e.g. a hardware power cut. That's the kind of atomicity that ZFS is concerned about—guaranteeing that changes that make it into the filesystem journal, are complete valid change units. In this case, one Lua program is one change unit in the journal.
Picture a database server where once you begin a transaction, the whole DB server goes single-threaded and doesn't serve any other clients until the TX completes. Fundamentally, at least for its more arcane global operations, that's exactly what a filesystem is. Usually this doesn't matter, because even these arcane filesystem operations take on the order of microseconds to complete (just with lots of non-locked pre/post execution overhead that inflates the total syscall time.) But a userspace program can do arbitrary-much stuff in a callback. It can even just get into an infinite loop, and never get back to the kernel. This is why there isn't such a thing as a system call with userspace callbacks! (Or any API equivalent to one—e.g. one with tx_begin + tx_commit calls, where tx_begin acquires a global kernel lock before returning to userspace, under the expectation that userspace call tx_commit to release the kernel lock.)
On the other hand, submitting a "callback program" as a whole to the kernel, allows the kernel to statically analyze the behavior of the "callback program" in advance; and only if it is determined to be "simple" (e.g. if it predictably halts after a short time as a static property), will the kernel issue the program a capability for actually running that "callback program."
This is how compute shaders work. This is how eBPF works. And this is (apparently) how this ZFS Lua thing works. They all do it for similar reasons: to ensure that the program is a complete, soft-realtime practical unit of work to be executing in some bounded-time context.
You do get atomicity from this, but it's not MVCC atomicity. It's more like Redis's atomicity: during the execution of a ZFS Lua program, all other clients making system calls against the filesystem will wait on acquiring the filesystem mutex, and so will block until the program completes. There's never a case where other activity might "interleave" with the Lua program's execution, and cause it to retry. There is no other activity. The only case in which the Lua program can actually fail (and so get rolled back), is if it dies/is killed in the middle of execution, due to e.g. a hardware power cut. That's the kind of atomicity that ZFS is concerned about—guaranteeing that changes that make it into the filesystem journal, are complete valid change units. In this case, one Lua program is one change unit in the journal.