kbr2000
Joined 8 years ago
Joined 8 years ago
I would say most interesting texts (articles, books, school, ...) should leave stuff up to the reader's mind to figure out. That's how someone really learns. Versus pre-baked stuff like television etc.
If something does not resonate at first that's pretty normal. You could still take it apart and start investigating words or concepts that ring no bell, for example: waves, interference, demoscene, owls, Feynman.
Enjoy! ;)
You seem to be referring to the selection of candidates of files to transfer (along several possible criteria like modification time, file size or file contents using checksumming) [2]
Rsync is great. However for huge filesystems (many files and directories) with relatively less change, you'll need to think about "assisting" it somewhat (by feeding it its candidates obtained in a more efficient way, using --files-from=). For example: in a renderfarm system you would have additions of files, not really updates. Keep a list of frames that have finished rendering (in a cinematic film production this could be eg. 10h/frame), and use it to feed rsync. Otherwise you'll be spending hours for rsync to build its index (both sides) over huge filesystems, instead of transferring relatively few big and new files.
In workloads where you have many sync candidates (files) that have a majority of differing chunks, it might be worth rather disabling the delta-transfer algorithm (--whole-file) and saving on the tradeoffs.
[0] https://www.andrew.cmu.edu/course/15-749/READINGS/required/c...
[1] https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_...
[2] https://en.wikipedia.org/wiki/Rsync#Determining_which_files_...
"A translation of Legat's article on Reis' invention was obtained by Thomas Edison prior to his filing his patent application on a telephone in 1877. In correspondence of 1885, Edison credits Reis as having invented "the first telephone", with the limitation that it was "only musical not articulating"." [1]
Fascinating stuff nonetheless, these inventors and their ideas... See also previous experimenters [2]
[0] https://en.wikipedia.org/wiki/Johann_Philipp_Reis
[1] https://en.wikipedia.org/wiki/Reis_telephone
[2] https://en.wikipedia.org/wiki/Johann_Philipp_Reis#Previous_e...
> You must have a filesystem, located on the /dev/sda4 device, mounted at /mnt/lfs.
The /dev/sda4 device represents the fourth (primary) partition on the /dev/sda block device, which represents the first SCSI disk.
It generates indices for its backups that allow you to search for files over all snapshots taken (which gives you an overview of which snapshots contain some file for you to retrieve/inspect). See dirvish-locate.
Does expiration of snapshots, given your retention strategy (encoded in rules, see dirvish.conf and dirvish-expire).
It consistently creates long rsync commandlines you would otherwise need to do by hand.
In the end you get one directory per snapshot, giving a complete view over what got backed up. Unchanged files are hard-linked thus limiting backup storage consumption. Changed files are stored. But each snapshot has the whole backed up structure in it so you could rsync it back at restore time (or pick selectively individual files if needed). Hence the "virtual".
Furthermore: backup reporting (summary files) which you could be piped into an E-mail or turned into a webpage, good and simple documentation, pre/post scripts (this turns out to be really useful to do DB dumps before taking a backup etc.)
You'll still need to take care of all other aspects of designing your backup storage (SAS controllers/backplanes/cabling, disks, RAID, LVM2, XFS, ...) and networking (10 GbE, switching, routing if needed, ...) if you need that (works too for only local though). Used this successfully in animation film development as an example, where it backed up hundreds of machines and centralized storage for a renderfarm, about 2 PBytes worth (with Coraid and SuperMicro hardware). Rsync traversing the filesystem to find out changes could be challenging at times with enormous FS (even based on only the metadata), but for that we created other backup jobs that where fed with specific file-lists generated by the renderfarm processes, thus skipping the search for changes...
This has been a lifesaver for 20+ years, thanks to JW Schultz!
The questions/topics in the article go really well along with it.
A PostScript program to visualize a calendar of moon phases (skip down to "LCAL PostScript Calendar Examples" for just that). Did some nice PS prints recently for the next 10 years, adapted to fit in a frame I had laying around.
[0] https://www.youtube.com/watch?v=UJaBspDXgzs [1] https://www.youtube.com/watch?v=Zc5uEwwr6S8
[0] https://www.lyx.org/
[1] https://en.wikipedia.org/wiki/WYSIWYM