shub

Sounds like they can do predicate pushdown to the storage nodes now. And...mainly beneficial for medium-to-highly selective scans of mostly uncached tables?

I am a dataologist, even if my title says software something something. Commercial software development is about getting data from point A to point B more than anything else. There's a little bit of logic, a little bit of munging, but almost all of what could be called an algorithm is done by a library or some other system--probably a database. It's maybe once a year that doing something algorithmic, e.g. reduce a regex to a hand-coded FSM, or write an exponentially decaying counter, is actually the best thing I can do to solve the problem at hand.

Reminds me of an IBM technical bulletin I found, somehow, about 10 years ago. It advised that if you were moving from this z9 hardware configuration to that one, you would have 30 minutes of downtime. Implying that other changes required no downtime, which I inferred wasn't stated because it wasn't insanely awesome to mainframe guys, adding CPUs and RAM to a running system was just how things worked.

I was in college at the time and looked at job postings for mainframe programmers because I wanted to work on that. Never found one that required less than 5-10 years experience, not then and not when I've checked every couple years after that. Too bad; I quite liked the idea of working in an ecosystem that starts from "let's make this work every single time" instead of "let's make this work well enough to keep customer complaints to a dull roar".

If you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA.

You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.

In addition, a lot of the techniques used to write high-performance Java boil down to "write it like C". Avoid interfaces, avoid polymorphic virtual calls (as you can't avoid virtuals entirely), avoid complex object graphs, avoid allocating as much as possible...it's not nearly as nice as naive Java. Still nicer than C IMO. If your process segfaults you can know for certain that it's a platform bug.

You're really stretching to find negative consequences there.

The short excerpt you can read for free says this is more about stuff like the two minutes between clocking out at closing time and locking the door on your way out. It might be wage theft in some sense but isn’t the same thing as requiring hours of unpaid overtime. On the other hand, I suspect the part of the article you can’t read for free is about an employer who tried to cover a substantial amount of work under this exemption and now the court has to ruin things for everybody to prevent abuses.

Why, on the off chance one flies into someone’s open mouth and the surprise triggers a swallowing reflex? I wouldn’t have a lot of sympathy for an adult who ate the homemade pills a guy threw at him and then suffered serious side effects.

Hanlon's razor?

That's why I took 102. They give you the straight dope there.

I always thought that force would work pretty well to build a dystopian hellscape.

Spicy. The fundamental problem is that a lot of people have jobs that pay a lot of money, I take it. You seem very upset about this.

On the other hand, I live far away and the YouTubes would be entertaining. Please do this, Google.

I guess they got the name back when smelling bad and smoking pot was progressive.