natekupp
Joined 15 years ago
Karma
79
Submissions
41
Member since
Oct 5
Recent submissions
Comment · 6 years ago
hey bserial, I'm part of the team working on Dagster.
While there are many things we're working on, there are 3 goals that got me excited about working on this system:
1. Local development: most modern workflow orchestration systems don't have a good local development story. We want to provide a seamless end-to-end dev experience from your laptop to CI to dev to prod for authoring data workflows.
2. Complexity: the Airflow deployments I've worked on or otherwise encountered have hundreds of DAGs and thousands of tasks scheduled on an hourly or daily cadence. We aim to provide abstractions to better support managing and wrangling that complexity.
3. Testability: Most modern data platforms are poorly tested. Many orchestration systems, like Airflow, tend to hardcode deployment concerns the business logic, e.g. EmrAddStepsOperator. With Dagster, we aim to separate the business logic from environmental concerns to make it easy to swap out an external resource implementation for a mock, dev version, etc.
Hope that makes sense!
Comment · 6 years ago
We use Pulumi to manage both our GCP and AWS resources, and we really like it.
You might consider using Terraform directly if you want something more mature.
Comment · 9 years ago
Thumbtack | Software Engineer, SRE, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.
Comment · 9 years ago
We use DynamoDB quite a bit at Thumbtack. Our biggest issue is backups - just wrote a short note about our experiences with DynamoDB here: https://medium.com/@natekupp/dynamodb-and-backups-16dba0dbcd...
Comment · 9 years ago
The original Science paper is here: http://science.sciencemag.org/content/356/6342/1046.full
The section "Relativistic deflections by foreground stars" walks through the math which enables this, which I found really interesting.
Comment · 9 years ago
Have you considered just applying through the jobs pages of the larger tech companies? Apple, Google, Facebook, LinkedIn are all on the peninsula and more accessible than startups in SF.
Comment · 9 years ago
We're also doing this at Thumbtack. We run all of our Spark jobs in job-scoped Cloud Dataproc clusters. We wrote a custom Airflow operator which launches a cluster, schedules a job on that cluster, and shuts down the cluster upon job completion. Since Google can bring up Spark clusters in < 90s and bills minutely, this works really well for us, simplifying our infrastructure and eliminating resource contention issues.
Comment · 9 years ago
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.
Comment · 9 years ago
I'm constantly surprised by how much work Amazon expects its customers to do themselves. The work that Segment has done here should be a service provided by AWS directly, continuously updating cost data in a Redshift database without any customer work required.
We just migrated our data infrastructure to GCP. One of the big motivators was experiences like this with AWS. We've got near-realtime GCP cost dashboards in BigQuery, and the only meaningful work on our end to make that happen was writing the SQL queries.
Comment · 9 years ago
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.
Comment · 9 years ago
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.
Previously Dagster Labs, Thumbtack, Apple.