Overview of a Storage Engine
A storage engine is the data plane component of the IO path of a persistent volume. In CAS architecture, users can choose different data planes for different application workloads based on a configuration policy. A Storage engine can be hardened to optimize a given workload either with a feature set or for performance.
Operators or administrators typically choose a storage engine with a specific software version and build optimized volume templates that are fine-tuned with the type of underlying disks, resiliency, number of replicas, set of nodes participating in the Kubernetes cluster. Users can then choose an optimal volume template at the time of volume provisioning, thus providing the maximum flexibility in running the optimum software and storage combination for all the storage volumes on a given Kubernetes cluster.
Types of Storage Engines
OpenEBS provides two types of storage engines.
- Jiva - Jiva is the first storage engine that was released in 0.1 version of OpenEBS and is the most simple to use. It is built in GoLang and uses LongHorn and gotgt stacks inside. Jiva runs entirely in user space and provides standard block storage capabilities such as synchronous replication. Jiva is suitable for smaller capacity workloads in general and not suitable when extensive snapshotting and cloning features are a major need. Read more details of Jiva here
- cStor - cStor is the most recently released storage engine, which became available from 0.7 version of OpenEBS. cStor is very robust, provides data consistency and supports enterprise storage features like snapshots and clones very well. It also comes with a robust storage pool feature for comprehensive storage management both interms of capacity and performance. Together with NDM (Node Disk Manager), cStor provides complete set of persistent storage features for stateful applications on Kubernetes. Read more details of cStor here
In the above figure:
SP - Storage Pool, the custom resource that represents the Jiva storage pool
CV - cStor Volume, the custom resource that represents the cStor volume
CVR - cStor Volume Replica
SPC - Storage Pool Claim, the custom resource that represents the cStor pool
CSP - cStor Storage Pool, the custom resource that represents each replica of cStor Pool
One SPC points to multiple CSPs. Similarly one CV points to as CVRs. Read detailed explanation of cStor Pools here.
Choosing a storage engine
Storage engine is chosen by specifying the annotation
openebs.io/cas-type in the StorageClass specification.
Sample spec - StorageClass for cStor
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: cStor-storageclass annotations: openebs.io/cas-type: cstor cas.openebs.io/config: | - name: StoragePoolClaim value: "cStorPool-SSD" provisioner: openebs.io/provisioner-iscsi
Sample spec - StorageClass for Jiva
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: jiva-storageclass annotations: openebs.io/cas-type: jiva cas.openebs.io/config: | - name: StoragePool value: default provisioner: openebs.io/provisioner-iscsi
When the cas-type is
jiva , StoragePool value of
default has a special meaning. When pool is
default , Jiva engine will carve out the data storage space for the replica pod from the storage space of the container (replica pod) itself. When the size of the required volume is small (like 5G to 10G), StoragePool
default works very well as it can be accommondated within the container itself.
Which storage engine to use ? (cStor vs Jiva)
Below table identifies few differences between the two engines.
|Important features: Synchronous replication, container native storage||Important features: Synchronous replication, container native storage, data consistency, capacity scaling, data resiliency, copy-on-write snapshots, instantaneous clones|
|Super Lightweight||Lightweight and superior read performance because of RAM as cache|
|Isolated IO stack||Shared storage pool|
|Ephemeral OS disks||SAS/SSD disks|
|Low capacity workloads||Any capacity workloads|
|Slow (Copy) clones||Faster and instantaneous snapshots and clones|
cStor is recommended most of the times as it commands more features and focussed development effort. cStor does offer robust features including snapshots/clones, storage pool features such as thin provisioning, on demand capacity additions etc.
Jiva is recommended for a low capacity workloads which can be accommodated within the container image storage, for example 5 to 50G. Jiva is very easy to use, and provides enterprise grade container native storage without the need of dedicated hard disks. Consider using cStor instead of Jiva especially when snapshots and clones capabilties are needed.