GitOps Issues: What to Expect and How to Handle the Challenges?

GitOps is the evolution in software development practices that started with the waterfall model and reached DevOps. GitOps improves upon the practices of DevOps to match with the needs and requirements of cloud-native development and infrastructure management. 

Things move fast in the software development world. As a result, it is difficult to keep abreast of all the development happenings. Sometimes GitOps is posited as the answer to all the challenges in production and deployment. But that is not true. Though GitOps has massive advantages, it also has its own set of challenges and limitations. In this article, let us take a look at some GitOps issues and how to overcome those challenges.

CI and CD separation 

GitOps’ principle is to separate CI and CD. In the DevOps model of CI/CD, deployment of tested code is the last step in the pipeline. But with GitOps, deployment is triggered with Git push and thus separating both functions. Practical implementation of this CI/CD separation is difficult to achieve. Deployment may need access to source code and all parameters rest with Git. The lack of GitOps tools to accomplish the separation also makes it difficult to implement. The major challenge is conducting smoke tests after deployment as Ops do not have visibility into source code. 

The CI pipeline handles the building and testing of code. A separate CD tool is employed for deployment. In the GitOps model, the CD can happen in two strategies: push-based deployment and pull-based deployment. A Git push is always a requirement for deployment in the first strategy. On the other hand, for pull-based deployment, the Kubernetes cluster will be monitoring for changes and applies the code changes themselves.

Multi-environment configurations

When your architecture requires multiple environments, GitOps makes it difficult to manage all the different environments. Since all the environment details are in the Git repository, each time the environment has to be changed a new Git push has to be executed. When there are a large number of production environments involved, creating different branches in Git becomes cumbersome, and tracking all the branches can go out of hand.

GitOps provides the methodology to set up infrastructure according to instructions in Git. Reusable code components can be used to replicate environments. This way setting up different environments can be automated. A metasystem can also be created that handles the different environments and automates the respective Git push to set up an infrastructure environment. This makes the creation of multiple environments easier and faster. Creating such a metasystem is critical when large-scale applications are developed that need multiple environments and quite often.

Limited observability

GitOps works with the basic level of abstraction, Git hash. It can only be comprehended by developers in the project. However, for project/product managers information at a higher level is much more useful. They are more concerned about the features in the latest build and will not be able to understand Git hashes and commits. One of the issues with GitOps is that it lacks this visibility. 

To obtain higher-level abstractions of data separate tools have to be built that read information from Git hashes. These abstractions can be displayed in a structured manner with visualizations. Such a tool is critical for large-scale applications as significant time would be lost trying to read the Git hashes for each Kubernetes. Moreover, a dashboard to view data is much more feasible.

Difficult to implement rollbacks

In principle, using a past commit helps for immediate rollbacks in GitOps as the complete history is stored in Git. But ambiguity due to multiple past commits can create problems in setting up the rollback process. Different software development teams will have different preferences for how rollbacks are implemented. GitOps tools largely do not have a standard practice for rollbacks. Automating rollbacks is also not well-defined.

There are multiple ways rollbacks can be implemented in an automated fashion even with GitOps. One method is to select a stable release as the default rollback option. Here, the same state is used even if there are stable working states released later. Another method is to revert everything to the last known working state. The rollback state is regularly updated with new stable releases. The second option is better. The state to fall back to will be tried and tested. Automated rollback procedures make the system resilient. 

Difficult promotion of releases

Large implementations have different environments to facilitate various testing and for redundancy. GitOps offers the path to deploy the code to an environment with Git commit. But it has no standard procedures to promote the code to different environments. One way to implement this is to have multiple commits for different environments. This creates the same problems as multi-environment configurations, Git overload. The general promotion path is from dev environment to test environment and then to production. Define pipelines to promote code automatically across environments to make promotion easier.

Cost overruns

GitOps is a very palatable solution for infrastructure automation. It makes developers’ life easier with easy methods to build, test, and deploy services for large applications. But one major factor of software development is part of the GitOps issues: the cost of operations. Using dynamically scalable architecture for different environments can cause the cost to spiral up. The primary concern for the developers is performance while costs are quite important for the business. GitOps being developer-friendly can let the cost rise dramatically and can go way over budget. This facet of software development is not addressed with GitOps.

Monitor costs regularly across infrastructure and projects. Analyze the past spending behavior to identify usage patterns and wastage of resources. The past data and current requirements can be used to forecast and budget for cloud spends. The practice of winding down resources when they are not used should be encouraged, especially for non-production environments.

Final words

GitOps as software development is promising and will be ubiquitous in the future. But GitOps in its current avatar has plenty of issues that are yet to be addressed. Most GitOps tools also do not address the challenges of GitOps implementation. Often development teams will have to rely on different tools to address each limitation. Microtica acts as one tool that offers solutions to many challenges faced in GitOps implementation addressing everything from scaling, monitoring, and costs.