Initial upload

otc/central-forgejo.think-ahead.cloud./stacks/local-backup/README.md

@@ -0,0 +1,126 @@
+# Local Backup with Velero and Minio
+
+This is example is adapted from the original icpbuilder stack.
+
+The two significant changes from the original were made:
+
+* disabled `hostPath` mount to persist backups within kind, since backups do not work sufficiently in this example due to PVC issues, see below.
+* renamed `minio` namespace to `minio-backup` so it does not collide with other minio examples.
+
+Within kind, it can only backup kubernetes objects. Data from PVC's is skipped, see below why.
+
+[Velero](https://velero.io/) requires some compatible storage providers as its backup target. This local installation uses [MinIO](https://min.io/) as an example.
+However, MinIO is not officially supported by Velero but works due to S3 compatibility.
+
+The current setup does NOT persist backups but stores them in MinIO's PVCs. Proper backups should configure external storage, see [Supported Providers](https://velero.io/docs/main/supported-providers/).
+
+## Installation
+
+The stack is installed as part of the `./example.sh` run.
+
+In order to persist a local backup you have to mount a local directory within `main.go`:
+
+```yaml
+nodes:
+  - role: control-plane
+    extraMounts:
+      - hostPath: /some/path/backup # replace with your own path
+        containerPath: /backup
+```
+
+Kind creates the directory on the host but you might have to adjust the permissions, otherwise the minio pod fails to start.
+
+## Using it
+
+After the installation velero and minio should be visible in ArgoCD.
+
+During the installation credentials for minio are generated and shared with velero. You can access them manually:
+
+```bash
+kubectl -n minio-backup get secret root-creds -o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
+# example output
+# rootPassword: aKKZzLnyry6OYZts17vMTf32H5ghFL4WYgu6bHujm
+# rootUser: ge8019yksArb7BICt3MLY9
+```
+
+A bucket in minio was created and velero uses it for its backups by default, see helm `values.yaml` files.
+
+
+### Backup and Restore
+
+Backups and subsequent restores can be scheduled by either using the velero cli or by creating CRD objects.
+
+Check the `./demo` directory for equivalent CRD manifests.
+
+Create a backup of the backstage namespace, see `schedule` task for more permanent setups:
+
+```shell
+velero backup create backstage-backup --include-namespaces backstage
+```
+
+There are more options to create a fine granular backup and to set the backup storage.
+See velero's docs for details.
+
+Check the backup with:
+```shell
+velero backup get
+```
+
+To get more details on the backup you need to be able to connect to velero's backup storage, i.e. minio.
+Using `kubefwd` here helps a lot (this is not necessary for restore).
+
+```shell
+kubefwd services -n minio-backup
+```
+
+More details with `describe` and `logs`:
+
+```shell
+velero backup describe backstage-backup --details
+velero backup logs backstage-backup
+```
+
+Restore the backup into the original namespace, you might want to delete the existing namespace beforehand:
+
+```shell
+kubectl delete namespace backstage
+velero restore create --from-backup backstage-backup
+```
+
+When restoring, velero does not replace existing objects in the backup target.
+
+ArgoCD does pickup on the changes and also validates that the backup is in sync.
+
+
+## Issues with Persistent Volumes
+
+Velero has no issue to backup kubernetes objects like Deployments, ConfigMaps, etc. since they are just yaml/json definitions.
+Volumes containing data are, however, more complex. The preferred type of backup are kubernetes' VolumeSnapshots as they consistently store the state
+of a volume at a given point in time in an atomic action. Those snapshots live within the cluster and are subsequently downloaded into one of velero's
+storage backends for safekeeping.
+
+However, VolumeSnapshots are only possible on storage backends that support them via CSI drivers.
+Backends like `nfs` or `hostPath` do NOT support them. Here, velero uses an alternative method
+called [File System Backups](https://velero.io/docs/main/file-system-backup/).
+In essence, this a simple copy operation based on the file system. Even though
+this uses more sophisticated tooling under the hood, i.e. kopia, it is not
+possible to create a backup in an atomic transaction. Thus, the resulting backup
+might be inconsistent.
+
+Furthermore, for file system backups to work velero installs a node-agent as a
+DaemonSet on each Kubernetes node. The agent is aware of the node's internal
+storage and accesses the directories on the host directly to copy the files.
+This is not supported for hostPath volumes as they mount an arbitrary path
+on the host. In theory, a backup is possible but due extra config and security
+considerations intentionally skipped. Kind's local-path provider storage uses
+a hostPath and is thus not supported for any kind of backup.
+
+## TODOs
+
+* The MinIO -backup installation is only intended as an example and must either
+be configured properly or replaced.
+
+* The current example does not automatically schedule backups.
+
+* velero chart must be properly parameterized
+