marinj

Joined 8 years ago

Karma

Submissions

Member since

Jan 23

Recent submissions

There are some complains that we can share, but honestly they are not really that fun to read. We are still early in our transition and one major complaint I have is about helm and lack of flexibility of it and operator being very flexible but a huge time sink.

I think a lot of complaints we would usually have are reduced due to the fact that we use managed K8s with GKE.

All other complaints I can think of are related to our application and some architecture decisions we made early on, but that won't be interesting to anyone but people at GitLab.

Comment · 6 years ago

> Wrong. Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes.

This is only one of the shards of one service that we migrated, and we migrated many more.

I did a high level writeup of this and many other things we've observed in https://about.gitlab.com/handbook/engineering/infrastructure...

if you are curious. I am happy to have a conversation with anyone about our experiences in more detail, not to convince anyone that k8s is the "one-true-way" (because I am still not convinced myself), but of the benefits this type of change can bring.

> They tried to use auto-scaling but failed completely and ended up disabling it

That issue is just one of many though. We disabled it at the time, but we have it enabled for a number of other services.

Regardless of how I personally feel about K8s, I have to say that the migration we are doing for GitLab.com is generating set of benefits that goes far beyond just moving to a new platform.

One of the largest benefits I've seen so far is that it was a great forcing function to resolve some long running architectural challenges, and is making us think more about how the application can run at a very large scale, without being at the very large scale. Things that we could get away before, like the issue you referenced there, we can't anymore.

Disclaimer: I am one of the people involved in this migration.

Comment · 7 years ago

> yet when they just dip their toes into kubernetes world, when they switched their docker image registry to run on kube, they've quickly discovered glaring omissions like missing liveness probe and storm of errors on any version update.

You are right, we had some missteps with the Helm chart that were unfortunately not discovered by others or us. Our test cases of scaling registry up and down worked perfectly in all our synthetic tests we did so it was not obvious to us that the liveness probe was missing. In hindsight it is quite obvious but at that time we had 16 other charts to write and some things did slip through. For a number of other services, community and paying users were reporting issues that we solved as we went further. Registry is one of the components that receives traffic in bursts so for majority of users this was probably never an issue or it was one of those "gremlin" moments.

For GitLab.com the things are quite different. We hold 2PB of data in docker images alone and there is continuous flow of traffic. For GitLab.com scale, none of the services we are porting to K8s have the luxury of the bursty traffic so we are careful in how and when we switch over traffic. The good thing is that all the edge cases we found are fixed and now in the Helm charts releases so that users can really put this on k8s as ready. If you are curious on all the issues we had to cover during this process, see the main registry migration epic https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/70 .

Comment · 8 years ago

GitLab.com is using the omnibus-gitlab package and it is configured in the HA configuration as per docs in https://docs.gitlab.com/ee/administration/high_availability/... . The actual update is being done using the directions in https://docs.gitlab.com/omnibus/update/README.html#multi-nod.... We use a custom built tool to enforce that zero downtime upgrade and the tool is in https://gitlab.com/gitlab-org/takeoff/ .

I do have to note that the tool is something we use temporarily as we are working towards refactoring part of infrastructure towards Kubernetes.

Comment · 8 years ago

I work for GitLab and I was responsible for introducing release changes for this cycle.

That is correct, we deploy our RCs to production. I consider RC as only a point in time snapshot of a release that will be sent to public.

One thing that is incorrect is that we push all our RCs to production. For example, this release RC1 did not get to GitLab.com because during deployment to our other environments we found an issue that could have caused a large problem at scale. However, we executed a lot of QA and FA tests against all our RCs. You can see this here https://gitlab.com/gitlab-org/release/tasks/issues?scope=all... .

If we only deployed a final release, we would most likely be overwhelmed with the amount of changes GitLab receives and it would be hard for us to monitor and limit the impact.

We are working on getting to continuous deployment to GitLab.com, and as part of this push we started right now with the tools we have at our disposal. One of the ideas to get to stable, non impactful deploys was to create many RCs. Thinking was that if we have a smaller delta between the RCs, we can more effectively check the changes and reduce the overall impact. We also wanted to make sure that we over-communicate our status updates so that we can get feedback in case we overlooked something.

Was it problem free? Nope. Did it limit the impact to users? I firmly believe so. We have a lot of work to do to get to blue-green deployments and continuous deployment on GitLab.com scale while we also continue to ship to on premise customers. I do believe that we will get there the best way we know how to, and that is iterating on changes.