How To Establish Better Governance
Who Owns This Workload?
Recently, one of our Site Reliability Engineers (SRE) noticed a workload running in our cluster which he hasn’t seen before. The workload was consuming some resources and the SRE wanted to apply some updates to the cluster but was not sure who owns the workload, and doesn’t know if his updates would have an impact on it or not. The SRE reached out to the development team on Slack concerning the Workload to see if anyone knew anything about this new workload. It was late in the day, and it took some time to get a reply. Many people did not know about this new workload, or who owns it, or what part of the system it belonged to. Some of us even wondered if this could be a malicious application. Should we terminate it and remove it right away? What if it is part of the new feature we launched a few days ago?
Eventually, we figured out that it was part of a prototype that an engineer was working on with someone from the business team to experiment with a new feature. But what if this was some malicious application? Or it was causing an issue in production and we needed to reach the owner of this service? Many critical hours would’ve passed without reaching a resolution. In such a distributed and decentralized environment and as your teams continue to grow, it becomes challenging to know everything and ensure everyone is a good citizen. So what should we do?
How To Establish Better Governance And Avoid This Problem?
If we mandate it that each new service or every change made to production should be reviewed by the SRE team, we will certainly slow the development team and slow the pace of innovation. Also, the manual review does not guarantee the detection of all violations. Therefore, the solution is to establish a governance framework and automate its process as much as possible in order for us to have the opportunity to scale.
There are three key elements that should be defined in order to establish good governance around the issue we faced. These elements are “targets”, “policies”, and “triggers”, as we explained in “Kubernetes Governance 101”. The target in this case would be all workloads in our cluster. We should apply the policy to all types of workloads: ReplicaSets, StatefulSets, Jobs, DaemonSets, etc. The policy would be to enforce that each workload has an “owner” label with the email of the engineer responsible for it. For the trigger, we thought initially that it is enough to run this once a day to find out who was violating this policy.
How To Setup A Custom Advisor In Magalix
Magalix offers KubeAdvisor which allows us to write custom policies using Rego - a declarative policy language that is open source and part of the Open Policy Agent project. This allows us to rely on a widely used framework and community-supported tools to make policy as code.
This is the Pathway within Magalix Console:
1. Navigate to KubeAdvisor
2. Create Custom Advisor
3. Create Issue
4. Write Rego Policy
This policy will check if the workloads spec (metadata.labels) contains an owner label and the value is not empty. If you are new to Rego, see the language reference
- Simply Browse Back to The “Issues”
Once the advisor and issue have been created, our system will run them within 24 hours. To see the violations, simply browse back to the “Issues” section under the relevant clusters. See more info about our console and how to navigate it to find violations and recommendations.
To Read More: https://hubs.ly/H0zlM_F0
Categories
- 10.2K All Categories
- 36 LFX Mentorship
- 89 LFX Mentorship: Linux Kernel
- 505 Linux Foundation Boot Camps
- 279 Cloud Engineer Boot Camp
- 104 Advanced Cloud Engineer Boot Camp
- 48 DevOps Engineer Boot Camp
- 41 Cloud Native Developer Boot Camp
- 2 Express Training Courses
- 2 Express Courses - Discussion Forum
- 1.8K Training Courses
- 17 LFC110 Class Forum
- 5 LFC131 Class Forum
- 20 LFD102 Class Forum
- 148 LFD103 Class Forum
- 13 LFD121 Class Forum
- 61 LFD201 Class Forum
- LFD210 Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- 23 LFD254 Class Forum
- 570 LFD259 Class Forum
- 100 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFS145 Class Forum
- 23 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 1 LFS203 Class Forum
- 46 LFS207 Class Forum
- 298 LFS211 Class Forum
- 53 LFS216 Class Forum
- 46 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 10 LFS244 Class Forum
- 27 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- 131 LFS253 Class Forum
- 998 LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 87 LFS260 Class Forum
- 126 LFS261 Class Forum
- 31 LFS262 Class Forum
- 79 LFS263 Class Forum
- 15 LFS264 Class Forum
- 10 LFS266 Class Forum
- 17 LFS267 Class Forum
- 17 LFS268 Class Forum
- 21 LFS269 Class Forum
- 201 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 212 LFW211 Class Forum
- 154 LFW212 Class Forum
- 899 Hardware
- 217 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 208 Networking
- 101 Printers & Scanners
- 85 Storage
- 749 Linux Distributions
- 88 Debian
- 64 Fedora
- 14 Linux Mint
- 13 Mageia
- 24 openSUSE
- 133 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 355 Ubuntu
- 473 Linux System Administration
- 38 Cloud Computing
- 69 Command Line/Scripting
- Github systems admin projects
- 94 Linux Security
- 77 Network Management
- 108 System Management
- 49 Web Management
- 63 Mobile Computing
- 22 Android
- 27 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 528 Off Topic
- 127 Introductions
- 213 Small Talk
- 20 Study Material
- 796 Programming and Development
- 264 Kernel Development
- 498 Software Development
- 923 Software
- 258 Applications
- 182 Command Line
- 2 Compiling/Installing
- 76 Games
- 316 Installation
- 54 All In Program
- 54 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)