GitOps - DevOps for Infrastructure Automation

GitOps offers a way to automate and manage infrastructure. It does this by using the same DevOps best practices that many teams already use, such as version control, code review, and CI/CD pipelines.

Companies have been adopting DevOps because of its great potential to improve productivity and software quality. Along the way, we’ve found ways to automate the software development lifecycle. But when it comes to infrastructure setup and deployments, it’s still mostly a manual process.

With GitOps teams can automate the infrastructure provisioning process. This is due to the ability to write your infrastructure as code (IaC) with the use of declaration files. We can store them in a Git repository, exactly as we store application development code.

How does GitOps work? 

The GitOps concept was initially introduced by Weaveworks, a Kubernetes management company. So discussions around GitOps are mainly in the context of Kubernetes. The transformation to microservices running in containers brought a need for orchestration platforms. Container-based applications can be complex and difficult for provisioning and management. GitOps helps in simplifying this by applying techniques proven in the DevOps world.

Nowadays the idea has become popular among DevOps enthusiasts, representing an upgraded model of the IaC concept. It revolves around 3 major components:

1. Infrastructure as code
2. Pull requests
3. CI/CD

Let’s look at them separately.

Infrastructure as Code

IaC is a practice of provisioning and managing infrastructure as declaration files, stored as code. By leveraging IaC and version control teams can optimize all operational procedures.

GitOps centers around the declarative model of IaC. This is why Kubernetes is a great example of implementation. Declarative means that configuration is more a declaration of an expected state, instead of a set of commands. For example, in Kubernetes, you can define the number of pods desired for a service in the manifest. The system will then take care of itself. No need for an engineer to write an imperative script that should get to the desired pod number.

Any cloud-native software that conforms to the declarative model can be treated as code. We use AWS CloudFormation, which is a declarative tool, to write AWS infrastructure. This means that we can treat infrastructure itself as code. Declare the desired state as code. The system applies the changes to achieve that state with automation.

With that said, declarative models are not a must to benefit in GitOps. You can do as well with imperatively defined environments.

Pull requests

The main idea behind the GitOps concept is that the version control system is a single source of truth. We use Git as a change management system for our application code. We can also use it for our infrastructure code. So the entire set of declaration files is in a single place where you can collaborate. This enables us to use the key concept of Git - the pull request for operational changes.

In an app development workflow, we use one main branch as a release branch. Developers create feature branches from the main branch. Develop a particular feature or story and when done create a pull request to merge it back into the main branch. This same approach is convenient for infrastructure code.

Creating a pull request enables the code to go through a process of code review before we integrate it into another branch of the codebase. Code reviews stop bad code from getting into test or production environments. This is even more important for infrastructure code. Having formal approvals in place via code reviews helps a lot with the auditing and troubleshooting.

Git organization

The deployment process in GitOps requires at least two repos: the application repo and the environment configuration repo. The first one contains the source code of the app together with its deployment manifests. The second one contains the desired state of the whole system described using a declarative specification for each environment. You can describe your environments as dev, test, production in a code repository, containing the applications and infrastructure services that can run with a particular version of that environment.

In the case of infrastructure, the main branch can represent an environment. We can implement the changes in the feature branch. Then create a pull request to merge the changes in the main branch. With this, we enable collaboration, while being transparent of who performed which changes. This is also beneficial for issue tracking to the root cause since all changes are commits in Git.

GitOps works with any Git-based system, like GitHub, BitBucket or GitLab. It is not dependent on any tool or technology.

CI/CD

To achieve a full GitOps implementation, you need a CI/CD pipeline. With automated delivery pipelines you can deliver infrastructure changes to designated environments, each time there is a change in the Git repository.
Pipelines are here to connect your Git pull requests to the orchestration system. When you trigger the pipeline with a pull request, the orchestration system executes the task.

There are two possibilities for a GitOps deployment strategy: Push and Pull Pipelines. The difference between them is in the way you ensure the deployment environment resembles the desired infrastructure.

Push Pipelines

Many popular CI/CD tools are using this strategy. We store the source code of the application and its deployment manifests in one repository. The build pipeline triggers when a new update happens in the application code. The pipeline builds the container images and pushes the changes to the environment. This strategy brings more flexibility, as it can support any type of infrastructure. The disadvantage is that it gives the CI/CD tool access to write to your environment.

Pull Pipelines

The community considers the pull pipeline approach a more secure practice for GitOps. With this approach, the operator is introduced. The operator is a component between the pipeline and the orchestration tool. It constantly compares the target state in the environment repository with the actual state in the deployed infrastructure. The operator changes the infrastructure to fit the environment repository if it detects any changes. Also, it's possible to monitor the image registry to identify new versions of images to deploy. This is what makes GitOps so special.

In GitOps the environment updates happen only when there are changes in the environment repository. The system reverts any modifications made if the implemented infrastructure changes in any manner not defined in the environment repository.

For most applications, you’ll probably need more than one environment. GitOps allows you to create multiple pipelines that can change the environment repository. You can use separate branches in the environment repository to manage more environments. The operator can react to the change of one branch by deploying to production and react to another branch by deploying to test.

What are the benefits of GitOps?

Using DevOps best practices

Since GitOps is a model focused on the pre-existing best practices of Git workflow, IaC, CI/CD pipelines, immutable servers, tracking, and observability, it represents a more advanced state of Kubernetes' cloud-native application management. Therefore, the current stack and experience within the company can serve a lot.

Continuous deployment—simplified

Continuous deployment means deploying faster and more often. Due to different considerations such as statefulness of systems, downtime resistance, upstream/downstream dependencies, and many other organizational relevant processes and dependencies, proper continuous deployment has been very challenging.

GitOps allows you to do this without having to manage a bunch of tools as everything occurs in the version control system. It provides structure and automation, thanks to the deployment operator.

This also increases productivity and faster MTTD (Mean-time-to-deployment). Automated continuous deployment ensures that the team can ship 30-100 times more changes every day, increasing average production performance 2-3 times.

Lower MTTR (Mean-time-to-repair)

MTTR is one of the key metrics DevOps teams should measure. Even small issues can be very challenging to repair in a microservice architecture. As GitOps keeps all changes in the version control system and the management is automated, it is possible to reduce MTTR significantly. You have a full overview of how the environment has changed and error recovery becomes quite easy.

Simplified Kubernetes management

Without getting to know Kubernetes thoroughly, developers can use familiar tools such as Git to handle Kubernetes upgrades and features more easily. Newly-embedded developers will get up to speed easily and be active in days rather than months.

Improved standardization across the company

You have transparent end-to-end workflows through the entire enterprise because GitOps has one framework for rendering applications, software, and Kubernetes add-on modifications. Git also fully reproduces your operations activities.

How to prepare for GitOps?

• Establish a stable code review and testing process. Carefully reviewing code changes can point out some obvious actions, like adding a global variable. It can prevent bad code from being released. You can then submit the validated code through pull requests, allowing no changes to be committed directly by developers. Once the pull request is reviewed and merged, you can trigger the pipeline. This is the first step in maintaining a high standard of code and subsequent stability of the system.

• Testing, testing, testing. Incorporating GitOps means having high-level automation that requires thorough testing of pipeline-released applications. Even though GitOps allows you to roll back relatively easily, releasing good code that has gone through good test cases makes your process more reliable.

• Focus on monitoring. GitOps allows repeatable operating processes, improvements in the traceable system state, rollouts, and rollbacks. Careful monitoring can help you identify and prevent any unintended drift and system changes in configuration. So, before starting with GitOps, review your monitoring skills and strengthen them in a way they can handle this change.

• Embrace the culture. Conventional process constraints with lengthy release times can only hold you back. Living a DevOps culture means leveraging the best strategies that will help a team understand the value of both development and operation actions. At the same time, they have to collaborate together to create an overall stable infrastructure, execute applications more rapidly and smoothly, and manage the system effectively. The lack of DevOps culture will prevent you from enjoying the benefits of GitOps.

Why GitOps?

GitOps is an extremely good workflow pattern that can help you efficiently handle cloud infrastructure. GitOps can provide an engineering team with numerous advantages, including better coordination, transparency, stability, and durability of the system.