dmw_ng

Fixed just for you :) I always forget that <meta> tag.

I'm a restic user, but have resisted the urge to attempt a bikeshed for a long time, mostly due to perf. It's index format seems to be slow and terrible and the chunking algorithm it uses (rabin fingerprints) is very slow compared to more recent alternatives (like FastCDC). Drives me nuts to watch it chugging along backing up or listing snapshots at nowhere close to the IO rate of the system while still making the fans run. Despite that it still seems to be the best free software option around

Modern .net on Linux is lovely, you can initialize a project, pull in the S3 client and write a 1-3 line C# program that AOT compiles to a single binary with none of the perf issues or GIL hand-wringing that plagues life in Python.

Given modern Python means type annotations everywhere, the convenience edge between it and modern C# (which dispenses with much of the javaesque boilerplate) is surprisingly thin, and the capabilities of the .net runtime far superior in many ways, making it quite an appealing alternative especially for perf sensitive stuff.

I recently bought an n100 and within a matter of days got buyer's remorse and impulse-purchased an n305 to go right beside it, which is currently sitting with a wildly overpriced 48 GB stick installed and 2TB SN850X, it's an absolute joy perfwise and the absence of heat it generates.

The only thing I'd reserve judgement on is the tendency to throttle. I haven't got far enough to characterize it, but it's not clear how much value those extra cores will add over the n100 with TDP settings tweaked down in the BIOS, and if leaving the n305 to run at max TDP, heat/noise/cost/temperature-related instability may start to become an issue, especially when packing other hot components like a decent SSD into the tiny cases they come in.

Seems like a massive distraction from their offering for a small company, wonder why they didn't consider something like tight integration with OnlyOffice or similar. Setting out to build a new office suite feels about as sensible as building a new web browser from scratch. Except at least with a browser, you have open specs helping you through most of the endless supply of compatibility problems.

> converting huge amount of xml files

> pickling

Sounds like if this is the tooling and the task at hand, about the most complex things that should be passing through the pickler are partitioned lists of filenames rather than raw data. E.g. you can have each partition generate a parquet for combining in a final step (pyarrow.concat_tables() looks useful), or if it were some other format you were working with, potentially sending flat arrays back to the parent process as giant bytestrings or similar

This is not to say the limitations don't suck, just that very often there are simple approaches to avoid most of the pain

It's comical to see the sudo codebase mentioned in the same breath as increasing auditability here

Sufficiently fast software often allows leaving out whole layers of crap and needless indirection, the most common being caching. Fixing an algorithm so you can omit a dedicated database of intermediate results can be a huge maintainability/usability improvement. The same principle appears all over the place, e.g. immediate mode UIs, better networking (e.g. CSS image tiling vs. just fixing small request overhead in HTTP1 vs. QUIC), importing giant CSV files via some elaborate ETL process vs. just having a lightning fast parser+query engine, etc.

Depending on how you look at it, you could view large chunks of DOM state through this lens, as intermediate data that only exists to cache HTML parsing results. What's the point of allocating a hash table to represent element attributes if they are unchanged from the source document, and reparsing them from the source is just as fast as keeping around the parsed form? etc. These kinds of tricks only tend to present themselves after optimization work is done, which is annoying, because it's usually so damn hard to justify optimization work in the first place.