Kubernetes is so cool and trendy! However there is always a question that remains... How do you backup PV when you have statefull application? Velero is a usefull tool that is able to backup your kubernetes ressource and also PVs.
Until now you had to specify Velero annotation when you want to backup pod volumes. But with the new version of Velero (1.5 and above) you can now backup PV by default and annotate resources when you don't want to backup the PV. So we change the paradigm here. In this article we will show you how up upgrade velero to 1.5 and switch your pod volume backup approach in order to backup all volume by default.
To summarize there is 2 approach:
opt-in pod volume backup approach
This is the only approach that exist for version 1.4 and below. Velero, by default, uses this approach to discover pod volumes that need to be backed up using restic, where every pod containing a volume to be backed up using restic must be annotated with the volume’s name.
Instructions to back up using this approach are as follows
kubectl -n your_namespace annotate pod/your_pod_name backup.velero.io/backup-volumes=your_volume_name,your_volume_name2
opt-out volume backup approach
This approach is available since version 1.5. In this approach, Velero will back up all pod volumes using restic with the exception of:
Volumes mounting the default service account token, kubernetes secrets, and config maps
Hostpath volumes
to exclude restic backup of volume pvc1-vm, you would run:
kubectl -n your_namespace annotate pod/your_pod_name backup.velero.io/backup-volumes-excludes=pvc1-vm
Upgrade velero and restic
Here is how to upgrade velero and restic from 1.4 to 1.5. First you have to upgrade the velero CLI, to do so just install the new version.
Get the latest version of velero CLI here. At the time of this article the latest version is 1.5.3.
untar and copy the binary to your bin path
tar -xvf velero-v1.5.3-linux-amd64.tar.gz
sudo cp velero-v1.5.3-linux-amd64/velero /usr/bin/
To check what is your CLI version:
velero version --client-only
output shoud be like
Client:
Version: v1.5.3
Git commit: 123109a3bcac11dbb6783d2758207bac0d0817cb
Next update the Velero custom resource definitions (CRDs) to include schema changes across all CRDs that are at the core of the new features in this release:
velero install --crds-only --dry-run -o yaml | kubectl apply -f -
Finally update the container image used by the Velero deployment and the restic daemon set:
kubectl set image deployment/velero \
velero=velero/velero:v1.5.3 \
--namespace velero
kubectl set image daemonset/restic \
restic=velero/velero:v1.5.3 \
--namespace velero
** Backup pod volume by default**
Okay so now your Velero is up to date and you want to change your approach for the opt-out in order to backup pod volume by default. To do so you need to edit the velero deployment to add a lauch argument
--default-volumes-to-restic
kubectl edit deploy velero -n velero
Go to spec.template.spec.containers.args and add this argument. See part of the yaml below.
... spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: deploy: velero strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: prometheus.io/path: /metrics prometheus.io/port: "8085" prometheus.io/scrape: "true" creationTimestamp: null labels: component: velero deploy: velero spec: containers: - args: - server - --features= - --default-volumes-to-restic #ADD THIS LINE command: - /velero ...
You need now to restart the pod. You can for example scale down the deployment and scale it up again.
kubectl scale deploy velero -n velero --replicas=0
wait pods to be removed..
kubectl scale deploy velero -n velero --replicas=1
Here we go you can now test it. For the example I will backup a whole namespace called gophish that contain a deployment with a PVC.
velero backup create my-backup --include-namespaces gophish
when it is done you can describe the backup
velero backup describe my-backup
The output:
Name: my-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.16.13
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=16
Phase: Completed
Errors: 0
Warnings: 0
Namespaces:
Included: gophish
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2021-01-20 11:05:10 +0100 CET
Completed: 2021-01-20 11:05:21 +0100 CET
Expiration: 2021-02-19 11:05:10 +0100 CET
Total items to be backed up: 25
Items backed up: 25
Velero-Native Snapshots: <none included>
Restic Backups (specify --details for more information):
Completed: 1
You can see at the bottom that a restic backup has been done. This is the pod volume that has been backed up without any annotation.
Enjoy!!
Feel free to comment this article if you have some question.
Source: velero.io/docs