Application Backups

For backups we are using standalone installation of Velero (opens in a new tab) with restic (opens in a new tab) on our Tanzu kubernetes clusters (info (opens in a new tab)).

Description

Velero creates a custom resource velero.io on kubernetes cluster, where it's stores all of it information about backups, schedules, restores, backup locations, etc. We are using S3 buckets as a storage, which were configured during installation and can be found in "velero.io/BackupStorageLocation" and can be listed by running

kubectl get backupstoragelocation --all-namespaces

Currently, we are running daily backups of only most important infrastructure, but this is subject to change in the future. All cluster with Velero installed and correctly configured backup locations can access all backups throughout all clusters. Meaning you can make a backup on "prod" cluster and restore it on "dev". This is possible since we are using restic (opens in a new tab) to actually make backups of files and persistent volumes. Restic uses generic format to store data and therefore our backups should be portable between cluster, different infrastructures or kubernetes version. This allows us, in case of emergency, to spin up an emergency cluster and restore entire infrastructure if needed.

The critical services are labeled critical=true, which you can use as selector when restoring a backups.

Installation

To use Velero, you need to install it's CLI client, which can be found here (opens in a new tab).
To validate if Velero is correctly install, you can run

velero help

which should list all available commands. By default velero will use your current kubeconfig and current-context cluster. To check if everything is working with our infrastructure you can run

velero get backups

which no matter what cluster you are connected to, should list the same backups.

As we are using S3 as backup locations, it is advisable to also install tools, that can work with S3 bucket, such as

• aws cli (opens in a new tab)
• s5cmd (opens in a new tab)
• S3 browser (opens in a new tab)

Usage

Here is a non exhaustive list of examples, how to work with Velero

💾 To make a backup with Velero

velero backup create app-namespace-backup-$(date -I) --include-namespaces my-app-namespace

which will create a backup of entire namespace my-app-namespace with name "app-namespace-backup-2022-01-01" (if it would be ran on date 2022-01-01).

🕛 To make repeating backup with Velero

you can create schedule (opens in a new tab)

velero schedule create app-namespace-backup --include-namespaces my-app-namespace --schedule="0 0 * * *"

which will create a daily backup, each day at 00:00AM, of my-app-namespace kubernetes namespace.

⏪ To restore backup with Velero

velero restore create my-app-restore --from-backup app-namespace-backup

or restore from schedule

velero restore create my-app-restore --from-schedule app-namespace-backup

which will restore the latest backup made from given schedule.
You can also use standard kubectl syntax and specify selector when creating a backup, restore, etc. For example, to restore only critical services from "backup123"

velero restore create --from-backup backup-123 --selector=critical=true

More information and examples can be found in

• VMWare Tanzu documentation (opens in a new tab)
• Official Velero documentation (opens in a new tab)