yachtman

It is similar to an article I read a while back saying we need "data engineers" not "data scientists". I think its generally true. Why my company, while it can do data science consulting, is choosing to focus on data engineering consulting first.

Sure thing. I won't here as that would be violating TOS for site

Whoa, sorry I think you added more emotional personal attack meaning to major fail than I intended. I major fail all the time, it's part of life and getting better.

No they do this because horizontal scalability is more general. Once you cross the threshold of what your meganode can handle you have to rewrite your code from the bottom up

The only way to do this is to build an integration that makes the switch DEAD SIMPLE. If it involves someone being highly motivated to switch over themselves without any help its not going to gain steam.

Disadvantage of retina screens, I didn't even realize. Thanks for pointing out, i'll fix.

Was SparkSQL ever intended to replace hive? My impression was that it was supposed to supplement spark for times it was convenient. I kind of suspected at one point they got caught up in the SQL hadoop race, but I always felt like it was best to do SQL elsewhere, and save spark for things that couldn't be easily expressed in SQL.

A company I worked at embraced nix. Beforehand I was a big docker fan and had few issues with it (yes there were occasionally caching issues and damn it docker figure out the issue with hyperkit on mac os, but largely its a productive tool). From an outside perspective nix just felt like alot more work, and nearly no one (except the people that set it up) could ever get it to work. So basically the entire build process for the core of the system was something that literally no one in the company wanted to touch. I'm sure if I was an insider on the tool I would appreciate the added stability you get on the backend, but at face value it seemed like it just made builds unapproachable on the front end. I'm happily back to using docker and haven't looked back.

Estimates for anything sufficiently complex are pure fantasy. They just serve to place an engineer in a bargaining conundrum and makes them implicitly decrease the scope for the 1000's+ decisions they will need to make during the course of implementation that they are not going to deliver the desired (also implicit) expectations. Its better to work from deadlines, then at least the engineer can try to hedge down the scope and deliver something they are comfortable with.

No one post any fork bombs...

I don't think that's how you define a bug. I think technically you could argue its a "logical error", however that assumes that you know the intent of the creator of bash was specifically to not allow this behavior which I don't think is the case.

Firefox is now objectively better than Chrome in every way. I would recommend switching moral reasons aside. Chrome's memory management is abysmal. I now get the same shudder when I see someone with chrome on their machine that I used to get when I saw someone running Internet Explorer (which may even be better than chrome now too).

I thought the general consensus was that the way that the craft was accelerating far outpaced any known tech by a large margin. I was hoping some aeronautics HNers could chime in with their 2 cents.