How To Establish Better Governance

ikordii

Who Owns This Workload?
Recently, one of our Site Reliability Engineers (SRE) noticed a workload running in our cluster which he hasn’t seen before. The workload was consuming some resources and the SRE wanted to apply some updates to the cluster but was not sure who owns the workload, and doesn’t know if his updates would have an impact on it or not. The SRE reached out to the development team on Slack concerning the Workload to see if anyone knew anything about this new workload. It was late in the day, and it took some time to get a reply. Many people did not know about this new workload, or who owns it, or what part of the system it belonged to. Some of us even wondered if this could be a malicious application. Should we terminate it and remove it right away? What if it is part of the new feature we launched a few days ago?

Eventually, we figured out that it was part of a prototype that an engineer was working on with someone from the business team to experiment with a new feature. But what if this was some malicious application? Or it was causing an issue in production and we needed to reach the owner of this service? Many critical hours would’ve passed without reaching a resolution. In such a distributed and decentralized environment and as your teams continue to grow, it becomes challenging to know everything and ensure everyone is a good citizen. So what should we do?

How To Establish Better Governance And Avoid This Problem?
If we mandate it that each new service or every change made to production should be reviewed by the SRE team, we will certainly slow the development team and slow the pace of innovation. Also, the manual review does not guarantee the detection of all violations. Therefore, the solution is to establish a governance framework and automate its process as much as possible in order for us to have the opportunity to scale.

There are three key elements that should be defined in order to establish good governance around the issue we faced. These elements are “targets”, “policies”, and “triggers”, as we explained in “Kubernetes Governance 101”. The target in this case would be all workloads in our cluster. We should apply the policy to all types of workloads: ReplicaSets, StatefulSets, Jobs, DaemonSets, etc. The policy would be to enforce that each workload has an “owner” label with the email of the engineer responsible for it. For the trigger, we thought initially that it is enough to run this once a day to find out who was violating this policy.

How To Setup A Custom Advisor In Magalix
Magalix offers KubeAdvisor which allows us to write custom policies using Rego - a declarative policy language that is open source and part of the Open Policy Agent project. This allows us to rely on a widely used framework and community-supported tools to make policy as code.

This is the Pathway within Magalix Console:
1. Navigate to KubeAdvisor
2. Create Custom Advisor
3. Create Issue
4. Write Rego Policy

This policy will check if the workloads spec (metadata.labels) contains an owner label and the value is not empty. If you are new to Rego, see the language reference

5. Simply Browse Back to The “Issues”

Once the advisor and issue have been created, our system will run them within 24 hours. To see the violations, simply browse back to the “Issues” section under the relevant clusters. See more info about our console and how to navigate it to find violations and recommendations.

To Read More: https://hubs.ly/H0zlM_F0