dan_manges

  aliases:
    common-env: &common-env
      key1: value1
      key2: value2

  tasks:
    - key: some-task
      run: ...
      env:
        <<: *common-env

Small scale: a project is almost small enough to run the build and tests locally, but you still want to have a consistent environment and avoid "works on my machine" problems.

Large scale: a project is so large that you need to leverage remote, distributed computing to run everything with a reasonable feedback loop, ideally under 10 minutes.

The opposite ends of the spectrum warrant different solutions. For small scale, actually being able to run the whole CI stack locally is ideal. For large scale, it's not feasible.

> A CI system that’s a joy to use, that sounds like a fantasy. What would it even be like? What would make using a CI system joyful to you?

I spent the past few years building RWX[1] to make a CI system joyful to use for large scale projects.

- Local CLI to read the workflow definitions locally and then run remotely. That way can you test changes to workflow definitions without having to commit and push.

- Remote breakpoints to pause execution at any point and connect via ssh, which is necessary when running on remote infrastructure.

- Automatic content-based caching with sandboxed executions, so that you can skip all of the duplicative steps that large scale CI otherwise would. Sandboxing ensures that the cache never produces false positives.

- Graph-based task definitions, rather than the 1 job : 1 VM model. This results in automatic and maximum parallelization, with no redundancy in setup for each job.

- The graph based model also provides an improved retry experience, and more flexibility in resource allocation. For example, one task in the DAG can crank up the CPU and memory without having to run more resources for downstream tasks (steps in other platforms).

We've made dozens of other improvements to the UX for projects with large build and test workflows. Big engineering teams love the experience.