Security policies in Kubernetes made easy

PodSecurityPolicy (PSP) is being deprecated in Kubernetes 1.21. There are several external admission controllers available like OPA/Gatekeeper, but getting started with OPA can be challenging.

Otomi, a complete suite of integrated K8s apps and add-ons, combined with developer self-service, is using OPA/Gatekeeper as the standard for providing policy enforcement. All apps integrated into Otomi are pre-configured based on sane defaults and best practices. Configuration of all apps is managed using a version control system where the desired state is reflected as code and the cluster state is automatically updated.

One of the key principles of Otomi is that it is easy to use and provides security from the ground up. In this article, I'll shortly explain how to get started with OPA/Gatekeeper using Otomi.

Install Otomi

First, install Otomi on a Kubernetes cluster in your cloud of choice. You can use the Otomi Quickstart to provision a Kubernetes cluster in AWS, Azure, GCP, or install Otomi using the Helm chart. To install Otomi using the Helm chart, make sure you have a Kubernetes cluster running with at least 16 CPU threads and 32GB+ RAM available in the worker node pool.

Add the Otomi repository:

helm repo add otomi otomi.io/otomi-core
helm repo update

Create a values file with the following values:

cluster:
  k8sVersion: '1.20' # currently 1.18, 1.19, 1.20 and 1.21 are supported
  name: # the name of your cluster
  provider: # choose between aws, azure, google or onprem

Install the chart:

helm install -f values.yaml otomi otomi/otomi

When the installer job (in the default namespace) has finished, copy the URL and the generated password from the bottom of the logs and complete the post-installation steps.

You can now log in to the console. In the left pane (under Enterprise) you will see Policies and when you click on it, you will see the list of all policies available in Otomi.

Managing policies with Otomi Console

Otomi contains a selection of 13 usable OPA policies and adapted them to be used by Conftest as well as for static analysis of manifests generated by Otomi.

You can turn policies on or off, and also set default parameters to be used. Select the policy you would like to edit (enable/disable or change default parameters), click Submit and click Deploy Changes. The changes will now be applied to the Otomi configuration.

Gatekeeper modes

Otomi supports 3 Gatekeeper modes:

• Enforcing
• Permissive (default)
• Disabled

In both Enforcing and Permissive modes, individual policies can be switched on or off. By default, Gatekeeper is enabled in permissive mode (logging & non-blocking).

Customization of the policies is supported based on the Otomi schema or by using Otomi Console. In case of specific requirements, operators can add their own custom policies.

To change the mode, open Gitea (in the Apps) and navigate to values/env/charts/gatekeeper-operator.yaml

To switch to enforcing mode, edit the file and set disableValidatingWebhook=false :

charts:
  gatekeeper-operator:
    disableValidatingWebhook: false

To disable policy enforcement, edit the file and set enabled: false:

charts:
  gatekeeper-operator:
    enabled: false

Developer support

In the team section in Otomi console, developers can use the Gatekeeper dashboard to see all the policy violations relevant to their deployments. First, create a Team in Otomi if you haven't done it already. In the team section, click Apps and then click Gatekeeper.

Running policy checks against any kind of YAML resource

With a simple command, you can test if a Helm chart is violating any policies.

The generated YAML files are streamed into Conftest and policies are tested one by one.

$ helm template stable/keycloak | conftest test --policy ./policies/ --all-namespaces -
FAIL - Policy: container-limits - container <keycloak> has no resource limits
FAIL - Policy: container-limits - container <keycloak-test> has no resource limits
162 tests, 160 passed, 0 warnings, 2 failures, 0 exceptions

By examining the log message, you can see that the container-limits policy is marking two resources as failures. Now, all you have to do is modify the templates to provide a "sensitive" amount of resource limits to the indicated containers and our policy checks will pass successfully!

This is pretty useful if you want to adopt new Helm applications, but don't want to deploy anything to the cluster unless it's well-examined for any violations. Conftest supports passing values to the policies using the –data option, which allows policy designers to configure different settings through parameters.

Policy exceptions capabilities

To create some flexibility, Otomi offers the ability to make exceptions by examining granular annotation information for every resource.

An example: using the following annotations will allow the entire pod or certain containers from the pod to exclude one or more policies:

# Annotation for the entire pod
policy.otomi.io/ignore: psp-allowed-repos
# Annotation for Istio sidecar based containers
policy.otomi.io/ignore-sidecar: psp-allowed-users,psp-capabilities
# Annotation for a specific container (name=app-container)
policy.otomi.io/ignore/app-container: banned-image-tags

Wrapping up

Implementing OPA/Gatekeeper can be a challenging endeavor. Some cloud providers have already integrated Gatekeeper into their managed Kubernetes service, but this results in a lock-in, while Otomi on the other hand is cloud-agnostic. Using integrated policies in a managed Kubernetes service is also not always very flexible. For instance, in Azure, policy enforcement can only be turned on or off. There is no option to run in permissive mode.

Otomi offers full Kubernetes security policies capabilities using OPA/Gatekeeper and Conftest. In the (default) permissive mode, developers can deploy their applications without having to directly comply. They can then see all the violations and adjust their deployments accordingly. With Otomi, you can have all OPA/Gatekeeper functionality with a sane set of implemented default policies, an easy user experience, and lots of flexibility, immediately available after installing Otomi.

To get started with Otomi today. Check out the GitHub repo here.