phiresky

Joined 8 years ago

Karma
3,459
Submissions
202
Member since
Jul 26
https://phiresky.github.io/blog/
View on Hacker News

Recent submissions

Comment · 2 months ago
Comment · 2 months ago
Which OS are you using? systemd-oomd (or similar) afaik is enabled in many modern releases and fixes it. The default settings (on Arch) make it kill stuff when your CPU is spending 60% of time on reclaiming memory over some time period. So you'll get 20 seconds of lag, then it is fixed.

By killing a program, yes, but how else do you expect it to work? If your RAM is not enough you have to kill stuff. And I really doubt you mean to say that Windows is _more RAM efficient_ than Linux, because that's kind of absurd. Maybe Firefox is less RAM efficient than Chrome.

Edit: MacOS I wouldn't doubt being better in RAM management than the others because they have been selling their devices with ridiculously low amounts of RAM for decades.

Comment · 2 months ago
Well I was assuming your comment was saying it's _impossible_ to do this without affecting output, so I was just describing an option on how it IS possible. I saw that token list biasing post and yes that looks pretty bad. I didn't look in enough detail to see whether it's actually clear what Anthropic and SynthID really do. I would assume though their approach is of the smart kind.
Comment · 2 months ago
Comment · 3 months ago
The EU told them they can not by default prefer their own maps product when linking from their search product, they have to allow the user to choose.

They could have simply added a selector when a user first clicks on the maps preview in the search result, and then remembered it on device or across that user's account.

But of course, then the user could choose a competitor's product and Google would have to honor it. That would be horrible, so instead they just made the user experience worse for everyone and elegantly made people blame the EU.

Comment · 6 months ago
I actually first thought this wasn't possible at all because I'm used to zstd which by default uses a 128MB window and I usually set it to the max (2GB window). 32kB is _really_ tiny in comparison. On the other hand though, zstd also compresses in parallel by default and has tools built in to handle these things, so seekable zstd archives are fairly common.
Comment · 6 months ago
I think you're looking at a different perspective than me. At _build time_ you need to process O(n), yes, and generate O(n) additional data. But I said "The amount of data you need to decompress is a constant". At _read time_, you need to do exactly three steps:

1. Load the file index - this one scales with the number of files unless you do something else smart and get it down to O(log(n)). This gives you an offset into the file. *That same offset/32 is an offset into your gzip index.*

2. take that offset, load 32kB into memory (constant - does not change by number of files, total size, or anything else apart from the actual file you are looking at)

3. decompress a 1MB chunk (or more if necessary)

So yes, it's a constant.

Comment · 6 months ago
I'm a bit disappointed that this only solves the "find index of file in tar" problem, but not at all the "partially read a tar.gz" file problem. So really you're still reading the whole file into memory, so why not just extract the files properly while you are doing that? Takes the same amount of time (O(n)) and less memory.

The gzip-random-access problem one is a lot more difficult because the gzip has internal state. But in any case, solutions exist! Apparently the internal state is only 32kB, so if you save this at 1MB offsets, you can reduce the amount of data you need to decompress for one file access to a constant. https://github.com/mxmlnkn/ratarmount does this, apparently using https://github.com/pauldmccarthy/indexed_gzip internally. zlib even has an example of this method in its own source tree: https://github.com/gcc-mirror/gcc/blob/master/zlib/examples/...

All depends on the use case of course. Seems like the author here has a pretty specific one - though I still don't see what the advantage of this is vs extracting in JS and adding all files individually to memfs. "Without any copying" doesn't really make sense because the only difference is copying ONE 1MB tar blob into a Uint8Array vs 1000 1kB file blobs

One very valid constraint the author makes is not being able to touch the source file. If you can do that, there's of course a thousand better solutions to all this - like using zip, which compresses each file individually and always has a central index at the end.

Comment · 7 months ago
There's a bit of an issue with the linked deployment (in my opinion). In the most zoomed out view you should see the first layer of blocks - very big blocks titled "English language", "French language", "German language". See https://phiresky.github.io/isbn-visualization/ maybe. That makes it a bit easier to read.

The point of the visualization is showing different attributes of books in the space of ISBNs. ISBNs correlate with country, publisher, and release date, that's why using it as a space is useful. You can clearly see the history of when blocks were created, which blocks are rarer than others (present in fewer libraries), and (on the AA hosting) which blocks are more present in AA vs not.

In any case though, yes ISBNs as spatial data are clearly not perfect. Do you have any suggestions that would order the 100 million data points better?

Comment · 7 months ago
Comment · 7 months ago
Netdata used to be really impressively minimal, performant, and packed with functions. Fully GPL open source. You ran one install command and it started a web-ui at localhost:19999 in a few seconds. The UI loaded instantly and had hundreds of graphs. You could tell the author was a single opinionated person obsessed with the maximum of monitoring with the minimum footprint.

It auto-detects many programs like docker, nginx and postgresql and automatically creates dashboards for them. It also has many dashboards about system internals I didn't even know were great to monitor, so it taught me a lot. For example, seeing a CPU pinned at 100% processing interrupts because of a network interface overload or having time frames with high IOwait during a SQL query clearly meaning there's some larger seq scans happening.

You also needed zero configuration, no login, etc.

Then they added multi-instance monitoring purely client side - the browser remembers other instance domains and links between them - pretty neat and completely uninvasive.

Then they introduced their cloud login, where you can monitor multiple instances remotely/together. They had a `--no-cloud` flag though if you did not want it. But by now they've removed that flag and they say patching out the cloud functionality is bypassing their license [1]. Some functionality is locked behind premium upgrades, and you get prevented from adding more than N metrics or M instances. It's still _possible_ to use netdata without going through their cloud but you have to go through a nag window every time you try to open the local UI. It's clear they don't want you to use it anymore, and I don't really feel comfortable about their default auto-updating local install any more either.

Now it's still impressive and useful, but it's much more an enterprise focusued tool than an "i have this server i want to monitor" tool.

Of course I understand they need to make money, but what used to be trivial to understand (hooks into everything in your system it can and opens a single port to display it) has become a whole huge integrated ecosystem and for me personally it's competing in the space where I'd probably rather spend the time to make a proper Prometheus/Grafana setup instead.

[1] https://github.com/netdata/netdata/discussions/17594#discuss...

Comment · 8 months ago
A $120M spend on AWS is equivalent to around a $12M spend on Hetzner Dedicated (likely even less, the factor is 10-20x in my experience), so that would be 3% of their revenue from a single customer.
Comment · 10 months ago
Even then you can just err.downcast_ref::<std::Io::Error>() though to get the underlying IOError, no?
Comment · 10 months ago
I'm happy to answer any questions! Nice to see this here again :)

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton