kaathewise

Polars relies on threading heavily, even when streaming files. And it appears that each thread loads quite a bit of memory. I've encountered OOM issues when incrementally reading Arrow IPC files which had very large batches. Fixed it by setting $POLARS_MAX_THREADS to 1, which amusingly also improved the performance on my very narrow task.

True!

Older numerical libraries such as cephes had to work with heterogeneous non-IEEE floating point implementations: you can still see remnants of VAX tests in the docs, for example. That, too, must've been quite a struggle

Yeah, those operations (+fma) are the only ones for which IEEE 754 mandates correct rounding.

Calculating correct rounding for other functions without hardware support is both challenging to implement and is often really slow. And that's before one gets into special functions. Or, even worse, inverses of special functions. Sometimes you might be lucky when those don't over/underflow around the edges of the domain

StageX does reproducible builds, so they are signed independently and can also be verified locally. I don't think it applies to Astral, but it's useful for packages with a single maintainer or a vulnerable CI, where there is only one point of failure.

But I also think it'd be nice if projects provided a first-party StageX build, like many do with a Dockerfile or a Nix flake.

The first programming language I learned was Java. And for us non-native speakers who didn't know English very well at that point public static void did indeed sound like a magic spell. It was behind both an understanding and a language barriers

Totally. A good chuck of the formats are just TSV files with some metadata in header. Setting aside the drawbacks, this approach is both straightforward and flexible.

I think we're seeing some change in that regard, though. VCF got BCF and SAM and got BAM

One of the problems is that a lot of bioinformatics formats nowadays have to hold so much data that most text editors stop working properly. For example, FASTA splits DNA data into lines of 50-80 characters for readability. But in FASTQ, where the '>' and '+' characters collide with the quality scores, as far as I know, DNA and the quality data are always put into one line each. Trying to find a location in a 10k long line gets very awkward. And I'm sure some people can eyeball Phred scores from ASCII, but I think they are a minority, even among researchers.

Similarly, NEXUS files are also human-readable, but it'd be tough to discern the shape of inlined 200 node Newick trees.

When I was asking people who did actual bioinformatics (well, genomics) what some of their annoyances when working with the bioinf software were, having to do a bunch of busywork on files in-between pipeline steps (compressing/uncompressing, indexing) was one of the complaints mentioned.

I think there's a place in bioinformatics for a unified binary format which can take care of compression, indexing, and metadata. But with that list of requirements it'd have to be binary. Data analysis moved from CSVs and Excel files to Parquet, and I think there's a similar transition waiting to happen here

Yeah, I found `anyhow`'s `Contex` to be a great way of annotating bubbled up errors. The only problem is that using the lazy `with_context` can get somewhat unwieldy. For all the grief people give to Go's `if err != nil` Rust's method chaining can get out of hand too. One particular offender I wrote:

   match operator.propose(py).with_context(|| {
    anyhow!(
   "Operator {} failed while generating a proposal",
   operator.repr(py).unwrap()
  )
   })? {

Which is a combination of `rustfmt` giving up on long lines and also not formatting macros as well as functions

Yeah, I presume so. At least that's what Swiss Tables do. The paper is focused more on the asymptotics rather than the real-world hardware performance, so I can see why they chose not to handle such edge cases

Ah, I see. Best of luck with that!

There are a few projects where I'd love to see a modern spreadsheet implementation. CryptPad comes to mind. They use OnlyOffice, which is quite featurefull, but takes awhile to load and isn't as responsive.