smueller1234

Here's another example of a similar but slightly different failure mode of the live defense: The department where I got my degree had a practice around PhD defense where there would be a prof each representing the four research pillars + the dean. It was well known that certain professors disliked each other enough that they would snipe each other's candidates. You'd just hope not to work for one whose enemies got picked for the panel. If you were a superstar, you'd pass, but if you were an okay candidate or someone strong but with stage fright, you be could toast even if you did good research.

The defense was only briefly about the actual thesis, then switched over to whatever research interest the committee members had, and they'd drill into their pet subjects. This was in physics, hardly politics forward normally, and at a very well respected university.

As another comment points out: that's nothing compared to high energy/particle physics!

The Pierre Auger Observatory certainly is a large collaboration for the astroparticle physics domain though. It's a big international collaboration.

Quick anecdote: my name (S Mueller) is not on that paper's author list because we had a rule that you had to be in the collaboration for a year before getting authorship. You stayed on for a year after leaving. Very reasonable! At the time I was nonetheless a bit bummed about missing out on the big Science paper. I guess I'm on the retraction though ;)

The timelines matter as well: They were working on EUV at Zeiss (who make the lensing/mirroring systems) already in 2005. That's about 20 years of development.

Even if it's success rather than money, you still have survivorship bias to contend with, so it's not really much of a helpful distinction.

Slight problem with that if you would like to live in a functioning, thriving democracy: democracy in the sense of "one person, one vote" requires or at least greatly benefits from a broadly educated population. It's not sufficient, but very likely necessary.

You're right -- the theoretical particle physicists at my faculty were using Mathematica very heavily when I was still in academia and maintained a dedicated compute cluster for it.

They really did not appreciate the debugging experience, but maybe that's improved in 15 years. :)

I realize you're making a general point about space/IO ratios and the below is orthogonal, no contradiction.

It's actually a lot less user-facing per disk IO capacity that you will be able to "sell" in a large distributed storage system. There's constant maintenance churn to keep data available: - local hardware failure - planned larger scale maintenance - transient, unplanned larger scale failures (etc)

In general, you can fall back to using reconstruction from the erasure codes for serving during degradation. But that's a) enormously expensive in IO and CPU and b) you carry higher availability and/or durability risk because you lost redundancy.

Additionally, it may make sense to rebalance where data lives for optimal read throughput (and other performance reasons).

So in practice, there's constant rebalancing going on in a sophisticated distributed storage system that takes a good chunk of your HDD IOPS.

This + garbage collection also makes tape really unattractive for all but very static archives.

I think Chips and Cheese is more like a fine replacement for realworldtech.com sans the toxic and highly educational and entertaining forums. Anandtech was much more accessible to the general tech public, but also more commercial and thus hit and miss on the content (no judgement intended, gotta eat).

Google's internal systems have been written against the Colossus semantics for many, many years and thus benefit from it's upsides (performance, cost efficiency, reliability, strong isolation for a multi tenant system, ability to scale byte and IO usage fairly independently, tremendously good abstraction against and automation of underlying physical maintenance, etc) while not really having too much of an issue with any of the conscious trade-offs (like no random writes).

On the other hand, if you've been building your applications against expectations of different semantics (like POSIX), retrofitting this into your existing application is really hard, and potentially awkward. This is (IMO) why there hasn't been an overtly Colossus based Google Cloud offering previously. (Though it's well publicized that both Persistent Disk and GCS use Colossus in their implementation.)

One of the reasons why it would be extremely hard to just set up or build CFS elsewhere or on a different abstraction level is that while it may look quite achievable to implement the high level architecture, there is vast complexity in the practical implementation side. The tremendous user isolation it affords for an MT system, the resilience it has against various types of failures and high throughput planned maintenance, the specialization it and its dependencies have to use specific hardware optimally.

(I work on Google storage part time, I am not a Colossus developer.)

Concur, Colossus is one of the examples where Google built what almost feels like magic technology. I work on Google Storage (among other things), and I've wished for a Cloud offering that exposes Colossus for years.

I don't know that it took "AI branding" to convince anybody. I think these workloads potentially enabled additional demand/market for such a product that may not have been there before.

One of the challenges with exposing native Colossus was always that it's just different enough from how people elsewhere are used to use Storage that there was a lot of uncertainty about the addressable market of a "native" Colossus offering. It's not a POSIX file system. Some of the specific differences (eg. no random writes) are part of what makes Colossus powerful and performant on HDDs, but it means you have to write your application to work well within its constraints. Google has been doing that for a long time. If you haven't, even if it's an amazing product, is it worth rewriting your applications or middleware?

Rapid Storage basically addresses this by adding the object store API on top if it (TIL from this thread that there's a lower abstraction client in the works as well).

Anyway, the team behind this is awesome. Awesome tech, awesome people. Seeing this launched at Next and seeing some appreciation on HN makes me very grateful.

4% of revenue is terrifying for large corporations.