OpenEBS is one of the most popular container attached storage system for modern applications running on Kubernetes. It enables SREs to easily deploy and manage stateful workloads by using Kubernetes as the control plane to maintain it's high availability.

OpenEBS project is sort of a family of different storage engines for different use-cases and requirements. Broadly, the storage engines can be Replicated or Local, the former suited for cross-node high availability and the later optimised for superior performance and some node-level availability guarantees.

Architecture 🏗

OpenEBS deploys its control-plane and data-plane engines that helps getting the benefits it has to provide for Stateteful workloads. It runs on any platform and can convert available storage to persistent volumes to be used by the stateful workloads. When you are using openEBS, you may be using one of data engines from Mayastor , cStor, LocalPv or Jiva. All these engines are designed to give you granular independence and benefits over choosing your storage types according to your storage needs. Storage engines like CStor and Jiva are optimised for availability in node-failure scenarios and can be conceptually thought of to be composed of two parts like many other Kubernetes components, the control plane and the data plane.

In this article, we will only be discussing "Jiva".

Let's come over to OpenEBSCTL which is designed to automate the complex, error-prone tasks which are needed to be done manually. It is a developing CLI supporting the Jiva upgrade jobs and many others for listing, describing and finding openEBS resources easily and nicely formatted. With the openEBSCTL, we have been trying to abstract away openEBS specific tasks so that the user is fully focused on their own workloads.

Something as challenging and critical as storage demands a well-defined upgrade path, it's best we proceed with caution. To make the process easier for users, the OpenEBS community came up with this guide to upgrade it and in the course of my LFX internship, I further reduced the complexity to do the same steps by firing a single command via kubectl-plugin😉

Let's get back to how jiva volumes can be upgraded. As seen in the component-diagram above, openEBS has its own control-plane and data-plane that helps manage volume and storage. For the volumes to be upgraded, both the planes must be upgraded. These volumes are upgraded in two steps.

1. Control Plane Upgrade
2. Data Plane Upgrade

It is important to understand that data-planes can only be upgraded successfully once control plane is upgraded, it's in nature😉

Control plane 🛫 🎆

Depending upon the installation method, the upgrade steps can be different, if you are in doubt feel free to ask the community members in #openebs of K8s slack.

You may be interested in choosing the particular version to upgrade from the available versions of openEBS releases. Alternatively, if you want to upgrade the openEBS to the latest stable version, you have to apply CRDs manually. e.g. Apply the following jiva-operator to upgrade the jiva control-plane to the latest stable version

kubectl apply -f https://openebs.github.io/charts/jiva-operator.yaml

All the charts are available here for your reference.

Your Control plane has been upgraded at this point

• While Jiva volumes will continue to work, the management operations like _Ongoing Pool/Volume Provisioning, Volume Expansion, Volume Replica Migration, Volume Replica Scaling will not be supported due to difference in control plane and pools/volumes version.
• Check for the REMOUNT env in openebs-jiva-csi-node daemonset, if disabled then scaling down the application before upgrading the volume is recommended to avoid any read-only issues.

Once the control plane pods are up and running, one can proceed with the data plane. Upgrade manually or automatically via the CLI which can be installed with kubectl krew install openebs

Data Plane 🧰 📂

Upgrading the data-plane is as simple as firing a single command and sitting for the jobs to be completed. As discussed, we will be using OpenEBSCTL for upgrading our data-plane components.

Note: You can also install openEBSCTL as an independent binary or as a kubectl plugin, there will no difference in the command at all, but for the sake of consistency I will be firing OpenEBSCTL as a kubectl plugin.

Get to know about the upgrade command

$ kubectl openebs upgrade --help
Upgrade Volumes, storage engines, and interfaces in openebs application

Upgrade OpenEBS Data Plane Components

    Usage: 
    kubectl openebs upgrade volume [flags]

  Flags:
  -h, --help                   help for openebs upgrade command
  -f, --file                   provide menifest file containing job upgrade information
      --cas-type               [jiva | cStor | LocalPv] specify the cas-type to upgrade
      --to-version             the desired version for upgradation
      --image-prefix           if required the image prefix of the volume deployments can be
                               changed using the flag, defaults to whatever was present on old
                               deployments.
      --image-tag              if required the image tags for volume deployments can be changed
                               to a custom image tag using the flag, 
                               defaults to the --to-version mentioned above.

_{--to-version may be something you should not use, ideally to-version is fetched from the control plane after you have upgraded it. It is meant for granular level control.}

Upgrade Jiva 🚵‍♀️ 🚴‍♀️

kubectl openebs upgrade --cas-type jiva

This will upgrade your volumes to latest version.

Let's take a peek inside 🕵🏻‍♀️👇

1. The command searches for the running jiva volumes by auto-detecting namespace being used by the volumes.
2. Fetches the required details from the openebs control-plane to upgrade the volumes to desired versions.
3. Checks all the other jobs in the cluster and detects if similar upgrade job is already in progress to avoid race conditions.
4. Remove failed jobs if present, and creates a new job configuration from the details available.
5. Downloads all the required upgrade images from the remote with the information in configurations.
6. Dry-runs the upgrade job on client and server side for any chance of pre-failure.
7. Creates an upgrade job in the namespace where openebs artifacts are running.

Checking the status of the upgrade Job 🤷🏻🧘🏻‍♂️

You are provided with the command to track the progress of the upgrade job with options of continuous streaming of container logs or instantaneous collected logs from the jobs.

To view the status of the upgrade job run:

kubectl openebs upgrade status --cas-type jiva

It shows all the progress logs from the running containers in the job and shows you the current status of it.

use --wait flag to start the streaming event for the logs to see your volumes upgrading in front of your eyes. It's as simple as:

kubectl openebs upgrade status --cas-type jiva --wait

Conclusion 🏌️🏌️‍♂️

So, that was just a bunch of things to do to upgrade your simple or complex workflows of openEBS Jiva. With the openEBSCtl, we are able to abstract away and handle complex problems automatically without getting the load over managing tasks and conflict issues in the kubernetes cluster.

This project was the part of Linux Foundation Mentorship Program and my experience working on it was truly wholesome with a lot of exposure over new CNCF tools and cloud native technologies.

Thanks ✍🏼 🙏🏼

Thanks to CNCF for sponsoring mentorship experience with Linux Foundation Mentorship Program. Thanks to my mentors Kiran Mova, Harsh Vardhan, Shubam Bajpai and openEBS team for helping out with the project. Thanks to the watching eyes for reading this articles, would love to get feedbacks and queries. Meet you on Twitter.