Comment Comment by unruthless Would Databricks solve this problem? https://databricks.com/They are essentially Apache Spark-as-a-service and have an API that allows you to submit a job on a cluster that you can configure to autoscale: https://community.cloud.databricks.com/doc/api/#jobs.JobsSer... https://community.cloud.databricks.com/doc/api/#jobs.Cluster... parent unruthless 10 years ago
They are essentially Apache Spark-as-a-service and have an API that allows you to submit a job on a cluster that you can configure to autoscale: https://community.cloud.databricks.com/doc/api/#jobs.JobsSer... https://community.cloud.databricks.com/doc/api/#jobs.Cluster...