sysguru2046
Joined 5 months ago
Joined 5 months ago
Do you consider Event-Source Commerce as part of this project?
Examples of Commerce events:
Catalog events - Product created, price changed
Inventory events - Quantith change
Order events - order created, cancelled
Routing events - order received, routing decision made
Shipment events - shipment created, tracking assigned
Customer events - Customer registered, address changed
Supplier events - Feed arrived
Taxonomy Events - Category created
Millions of files, Interrupted transfers, Backup Server Reboots, Long-term operation?
My testing went well. I was able to mount ZeroFS as NFS and write a 100M file to the mount point. The file persists after I deleted ZeroFS cache.
My plan is to use it for backup process. If we don't have such tool, we would need to prepare a large Filesystem stage area, then run rclone to upload to cloud space. With this tool, I can mount it on Backup server, rsync file server a large Filesystem to the ZeroFS mount point, eliminating stage area.
Will ZeroFS support large Filesystem size about 800GB?
It inspects my existing repo, and identified existing logic, then imported my existing functions, and created a checkpoint_manager.py.
The new function adds additional functions like: - update_started - update_completed - update_failed - install_shutdown_handlers - restore_shutdown_handlers - persist_checkpoint
Below points are generated in Implementation Tasks:
3. Implement checkpoint model: - typed checkpoint structure; - schema version validation; - bounds validation for next_url_index. 4. Implement atomic checkpoint writer: <- "atomic" - write temporary file beside final checkpoint; - flush and fsync; - atomic rename; - preserve old checkpoint on failure. 5. Implement checkpoint loader: - handle missing file; - reject malformed JSON; - reject unsupported schema version; - reject URL fingerprint mismatch.
I don't fully understand what "atomic checkpoint writer" means, but the LLM understands what it means. The functions are likely to implement "atomic checkpoint writer".
The key takeaway is that tokens generated by human usually not restrict enough. When I thought about "checkpoint", I wasn't thinking of any "atomic" shape of checkpointing. I was using my casual language model to generate first version of implementation first. Then using Spec-Driven-Development model, the generated spec has better understanding of what it is. There could be many mutated forms of real life implementation of "checkpoint", the LLM generated code is mostly restrictive one that I will never know in my whole life experience. The LLM training must have undergone very restrictive learning process that I never experienced.