JoachimSchipper

Nice to see more people getting nerdsniped by the sh code. ;-)

Yes, associative arrays work well. I think it should even be possible to use bash associative arrays. But at that point you're no longer doing classic sh - awk is basically halfway to Perl. (And pretty awesome.)

The sort is on line numbers, as aozgaa said. Again, the paradigm here - and this is designed for a different time - is that your data most definitely does not fit in RAM, so you use sort(1) to sort on disk and run your software using only constant memory. (In modern software, databases can and do sort on disk, but few programs do.)

In detail, for input "foo bar FOO qux FOO foo", we convert to

[1 foo, 2 bar, 3 foo, 4 qux, 5 foo, 6 foo]

(with newlines instead of commas, obviously), then sort by word (then line number) to

[2 bar, 1 foo, 3 foo, 5 foo, 6 foo, 4 qux]

at which point the uniq invocation gives <count> <first_line> <word>, i.e.

[1 2 bar, 4 1 foo, 1 4 qux]

albeit with an ugly mix of tabs and spaces. One final sort by <first_line> gives us

[4 1 foo, 1 2 bar, 1 4 qux]

and then it's just a matter of formatting the output:

[foo 4, bar 1, qux 1]

The generally-useful point is that the classic shell utilities really do work pretty well if you're operating within their paradigm, which isn't "throw everything in a hash table". (That's the paradigm of later scripting languages.)

The general point is true, but the shell pipeline gets a lot more elegant if you use the sort-and-accumulate paradigm that the classic shell utilities were written for (which uses O(1) memory, by sorting on disk). Using mostly the author's own code, and adding --count to uniq:

    tr < README.md --complement --squeeze-repeats '[:alpha:]' '\n' \
    | tr A-Z a-z \
    | nl \
    | sort --key=2,2 --key=1,1n \
    | uniq --skip-fields=1 --count \
    | sort --key=2,2n \
    | awk '{ print $3, $1 }'

(Where the final awk papers over the fact that we're mixing tabs and spaces here; obviously, awk is also good at doing the accumulation step, but uniq --count suffices here.)

(I originally posted the above as a comment on lobste.rs, on this same article.)

In a way, but note that the government actors here have Opinions on the Chinese having capable models.

Thanks a lot for your response!

You're commenting on an article about people consuming less than $3/day; Americans on food stamps (SNAP appears to be about $4-6/day alone, not counting any other benefits) are a distraction, simply not part of the population that the article is discussing.

I really like this article. I do think it's useful to consider that the unit of isolation ("process") of the cloud era is a VM or container, and that the major clouds do have some sort of permissions model.

marktplaats.nl is the big "eBay-like" for the Netherlands.

It's not ideal, but it's not a cryptographic risk.

Using an encoding that (like Protobuf) has multiple representations for a message may cause you problems if you switch implementations - sha256(encode(msg)) might yield different hashes on different implementations of encode().

But the main risk is an encoding that has multiple interpretations of a single encoding (e.g. sha256(encode("admin", "true")) == sha256(encode("admint", "rue"))), and Protobuf (can be unserialized, and thus) doesn't have that problem.

I was also thinking of promise pipelining, but note that the article proposes communicating A -> {B, C} -> D, never directly communicating from A to D. Cap 'n Proto "level 3" nodes could send promises from B and C to D, but that still needs A to talk to D, i.e. A -> {B, C, D}; {B, C} -> D. Same latency / depth of dependency chain, but still more messages - right?

(In return, note that Cap 'n Proto's A -> D message makes it more obvious how A figures out whether the operation succeeded; I'm not quite sure how that works in the proposed diagram. I suppose the proposed system actually puts all messages in a system-wide database, which does solve the problem.)