MichaelSalib
Joined 16 years ago
Karma
2,425
Submissions
728
Member since
Apr 14
Recent submissions
Comment · 6 years ago
Comment · 6 years ago
To be clear, our experience using gcsfuse and friends to do basically the same things was extremely painful and a performance nightmare. The HDF format was designed for a world where seeks are free which makes cloud access very high latency and very low throughput.
Comment · 6 years ago
Ah, thanks for these! But I see nothing has changed.
* pyfive is interesting but immature and doesn't seem to have any cloud bucket support
* h5s3 is an abandoned experiment that hasn't been touched in two years
* h5py is fine but again, no cloud support
* kita is a commercial offering from the HDF Group and -- I cannot stress this enough -- these people are shockingly incompetent; plus when I last looked at their system architecture diagram I thought it was a joke (well, I thought it was an intentional joke)
Efficient access to scientific datasets hosted on S3/GCP is a full blown crisis in the scientific computing community. People aren't switching to zarr for the fun of it, but because zarr is here, today, and isn't a joke, and is actually open.
Comment · 6 years ago
No, you can certainly write in parallel, despite the GIL. The GIL makes this inefficient if your work is CPU-bound, but for IO-bound workloads it can be fine.
Comment · 6 years ago
Otherwise, HDF5 offers every single advantage that zarray has and is much more mature, stable, better documented, and has better support.
Absolutely not. HDF5 is an awful format with terrible implementations. For example, try writing a python program with multiple threads where each thread writes to a different HDF5 file. This should just work -- there's no concurrent access. And yet it doesn't because HDF5 implementations are piles of ancient C code that use lots of global state. There's no technical reason for this; one could easily store all the state needed in a per-file object. But back in the day, software eng standards were lower (especially for scientists) and HDF5 changes at a glacial place.
I've been bitten by this particular bug, but you really have to wonder: given how poorly it speaks to the software engineering behind HDF5 implementations, what else is broken in the code or specifications?
If you're working in a situation where it makes sense to have things on disk or some sort of NFS share, use HDF5. If you're working with objects in a cloud bucket, you'll incur additional overhead with HDF5, as you'll have to read its table of indices, then make range requests to each chunk. Zarr is optimized for the cloud use case.
When last I looked, there were no open source HDF5 implementations that were smart enough to do range requests to cloud hosted hdf files. Has this changed?
Comment · 6 years ago
What's the plan for when grandma develops dementia? The paranoia will make her unwilling to part with her beloved firearm...do we just wait until she murders a family member or home health aid?
That seems like a much more important question than bickering over precisely which firearm grandma should have as she slides into dementia.
https://khn.org/news/dementia-and-gun-safety-when-should-agi...
Comment · 7 years ago
Which do you think looks better on your promotion packet?
1. Write a new libc and get it used all over Google, generating lots of users automatically; or
2. Shepard Google's adoption of code some other folks wrote.
Comment · 7 years ago
The idea that children should spend their days staring at screens and never interacting with other human beings seems really sick. But oh so on brand for the Valley.
Comment · 7 years ago
I worked at FB briefly so maybe I can explain. FB has a corporate culture that really discourages critique. When things are broken, especially internal things, people look at you funny if you speak up about it. A big part of that is that quarterly bonuses are given for "making an impact" and your group's status (and part of your bonus) is based on delivering a consistent set of "impacts" over time. So it is better for your comp to do things badly really fast since you get (1) did something super fast! and (2) get to record a big impact a few months later when you fix the obvious brokenness.
Pretty quickly, people learn to keep their mouth shut.
Also, many, many FB engineers are early-career folk who are fresh out of school. More senior folk are few and far between and are even more strongly incentivized to keep their mouth shut, because their bonuses are bigger.
Comment · 7 years ago
The way that white supremacy works is that black people are penalized even in the small enclaves where they constitute a local majority. Note that the laws that punish crack (used by lower income black people) 100 times more harshly than powder cocaine [0] are federal in nature.
[0] https://www.aclu.org/other/cracks-system-20-years-unjust-fed...
Comment · 7 years ago
> When it comes to exceptions, it's generally true on x64 that you don't pay for what you don't use (there's no performance penalty to exception handling if you don't throw) although that hasn't always been true for all platforms and implementations. It's also generally true that you couldn't implement that kind of non local flow control more efficiently yourself, although the value of that guarantee is a little questionable with exception handling.
I'm inclined to agree with you but just about everyone at Google says the opposite and most C++ shops I've seen agree with them. I've made this argument and lost repeatedly. So, it seems like the community can't even agree on which abstractions are zero cost (or maybe whether some zero cost abstractions are actually zero cost?). To the extent that the community itself has no consensus about these things, maybe they're not a marketing slogan that's helpful to use.
Comment · 7 years ago
(1) I love your writing.
(2) Even reducing it to runtime costs, it seems a bit nonsensical. Are C++ exceptions a zero cost abstraction? All the googlers I argued with about them would insist that they have unacceptably high runtime costs.
OK, but templates are surely zero (runtime) cost abstractions, right? Unless you start to worry about duplicate code blowing out your instruction cache but if that's a problem, no profiler in the world will ever be able to tell you, so I guess you'll never know just how costly the abstraction is, so you might as well continue believing it is zero...?
Comment · 7 years ago
Zero-cost abstractions only exist in a world where you don't highly value language simplicity and comprehensibility.
Simplicity and comprehensibility were things the committee had to give up in order to pretend they had "zero-cost" abstractions. Nothing in life comes free: everything, including all abstractions, comes at some cost.
Comment · 7 years ago
To clarify: of course they like simplicity when it costs nothing. But they consistently value other goods over simplicity.
For example: maintaining backwards compatibility. The community believes that it is more important that 20 year old C++ code run unmodified than that the language should be simplified. There's lots of stuff you could do to simplify the language but options dry up in a world where 20 year old code must be able to run unmodified.
So sure, the committee talks a lot about simplicity, but it isn't willing to sacrifice much.
Don't get me wrong: I'm glad that finally, in 2020, C++ will be almost but not quite as good as Common Lisp was at metaprogramming back in 1982. But it remains the case that eval-when and defmacro are both more powerful and dramatically simpler than anything the C++ committee has ever considered.
Comment · 7 years ago
I think that's dependent on language culture. Because the C++ community doesn't take language simplicity or comprehensibility seriously, there are lots of C++ developers who can't use or reason about surprisingly large parts of the language. So the community has rallied around the notion that "library" developers need to understand everything and that most developers will just glue together bits that the library devs made.
I mean, how many C++ developers actually write serious template code? How many of them could reliably explain what the keywords in post do?
The idea that every developer is a library author (or the lisp extension that every developer is a language author) is common in many other language communities but it relies on the community working hard to make mastery of the language feasible for lots of people. The C++ community never bought into that notion; they inherited a very stratified class structure from Bell Labs.
This is all completely wrong.