Comment by bpodgursky

I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern).

I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.