We had python scripts that generated GitLab CI/CD yaml [1]. Tooting my own horn here, but it was super cool to ship fairly fast for the first year or so. By the end, we had something like 5 MB of yaml, but in order for the GitLab SaaS backend to process it, it took something like 32 gigs of ram on their MergeRequestProcessor SideKiq worker.
They had to open a whole epic in order to reduce the memory usage, but I think all that work just let us continue to use GitLab as the number of services we grew increased. They recommended we use something called parent/child pipelines, but it would have been a fairly large rewrite of our logic.
They had to open a whole epic in order to reduce the memory usage, but I think all that work just let us continue to use GitLab as the number of services we grew increased. They recommended we use something called parent/child pipelines, but it would have been a fairly large rewrite of our logic.
[1]: https://docs.gitlab.com/ci/yaml/