Three of the critical web services used by the MUSES collaboration are Discourse (this forum), Nextcloud (cloud file storage and calendar/groupware solution), and HedgeDoc (real-time collaborative documents). These services should be resilient against data corruption and other failure modes.
The MUSES web services are backed up in multiple ways, depending on the persistent volumes backing the service data.
longhorn) volumes are backed up using the native backup system configured via the Longhorn web UI. The target location of the backups is
radiant-nfs.ncsa.illinois.edu:/radiant/projects/bbdr/muses/backup/backupstore, but these are not raw files; they must be restored via Longhorn.
NFS-mounted NCSA Condo storage (
nfs-condo) volumes are backed up manually to
Backups stored in the NFS-mounted NCSA Condo storage under
radiant-nfs.ncsa.illinois.edu:/radiant/projects/bbdr/muses are accessed by opening an SSH terminal into a worker node and mounting the directory via NFS. The SSH access requires adding your SSH public key to the
authorized_keys in the cluster nodes and the commands below assume an SSH config like so:
$ cat ~/.ssh/config.d/muses
# Automatically created by terraform
SSH into a worker node for example as shown below and mount the Condo volume via NFS. The subsequent commands assume a base path
$ ssh muses-worker-0
$ mount radiant-nfs.ncsa.illinois.edu:/radiant/projects/bbdr/muses /mnt
I created a backup system with a Helm chart that can be added as a dependency to the Helm charts of the deployed services, allowing us to enable backups by sprinkling in some lines to the relevant
values.yaml file like this example from our Discourse chart:
Read more about how it works in the backup chart’s Readme, including how to enable the restore deployment for convenient restoration of backups.
The backup locations listed below assume a base path
Discourse has a native backup system that generates nightly
.tar.gz archive files of the entire instance. To restore these you install a fresh deployment and then restore from one of these files. To ensure these backup files are available in the event the entire deployment and associated PVCs are deleted, the backups chart is used to take snapshots of the Discourse data. This is redundant in the sense that it is snapshots of snapshots.
An example of a backup file location is
Nextcloud and HedgeDoc both use the common pattern of having a flat file data volume and a SQL database volume. The backups chart handles this common configuration for both MySQL and PostgreSQL databases.
Nextcloud backups are found in folders with the patterns
/nextcloud/nextcloud-data/snapshot.Xfor the flat files and
/nextcloud/nextcloud-db/YYYYMMDDfor the database dumps.
HedgeDoc backups are in analogous locations.