dundun

How does this relate to Zipline and Bighead? Does it replace those projects or is it a continuation of them?

There's a ton of dunning-kruger going on, but I think as college kids riding a high of winning a contest and learning a ton, it can be excused.

Google opensourced Tensorflow because they believed it would help with the hiring process: if researchers could use the same framework to do their PhDs as Google used in their production systems, that was seen as an advantage.

Maybe that's Meta's play here? Maybe the idea is that the ecosystem around a model could be as valuable or more valuable than the model itself too, so an OSS model could benefit Meta a lot more by gaining more of the ecosystem mind share?

Or Maybe Yann LeCun is just a hippie that dreams of free love, hard drugs and open-source models?

There was also a breaking story about Google acquiring Spotify on the exact same date a year later!

Nothing in this move should affect contributions of Snakebite and Luigi. If anything, it will just make them easier to use with cloud environments in addition to bare metal.

With over 20K jobs/day, Hadoop will be a part of Spotify's data processing stack for quite a while. BigQuery is just a (awesome) piece of the full puzzle.

[spotifier]

Browsing through the code, this looks a bit rough around the edges and untested. I'd wonder how it compares to similar packages for Spark, Hadoop or Flink. Lack of numbers and users make me wonder if it's anything more than someone's PhD project at CMU.

MR is actually growing at Google still and heavily used by legacy systems and experienced engineers.

Compared to newer frameworks like those described in the FlumeJava and MillWheel papers, MR's growth is flat.

This is also happening in the Hadoop ecosystem too: if you're writing JavaMR by hand, you're probably spending more time and writing less efficient jobs than what you might get by an optimized pig/hive job with tez under the hood. Or through something in the Cascading or Crunch family, which provides useful abstractions on top of MR or other execution engines.

Then there are also a lot of tools popping up that take some if the use cases that were shoehorned into MR which are more natural outside, like ML/iterative computation through Spark.

MR isn't dying inside or outside Google, it's just being abstracted away.

What is missed in the article and many of these comments is that Hadoop isn't always going the best tool for one job. It shines in its multitenancy- when many users are running many jobs-each developed in their favorite framework or language(bash/awk pipeline? No problem) running over datasets bigger than single machines can handle.

It also comes in handy when your dataset grows dramatically in size.

Then you're hardly using commodity hardware anymore. While jobs like that probably actually work on Hadoop, I'd imagine a problem like that might be better suited for specialized systems.

A possible way to look at Airbnb is that they have 200k properties and 200k contractors that get paid a percentage of the revenue they generate.

They aren't in the hotel business, they're marketers and matchmakers.

Sorry, I should have clarified that comment before posting it.

Many bands allow audience taping but I'm not aware of m?any that allow for commercial distribution of those recordings. If Spotify attempted to distribute those recordings against the bands' policies then it would probably piss off the bands.

That Danny Schmidt is pretty great stuff. You're right about Spotify's lack of content when it comes to stuff like this-- they have all his studio albums but are kind of weak when it comes to live stuff, they have one album. Archive.org definitely fills in this gap (I use it all the time for Dead shows).