This page will walk you through the main steps required to manually back up and restore the state of a Pachyderm cluster in production. Details on how to perform those steps might vary depending on your infrastructure and cloud provider / on-premises setup. Refer to your provider’s documentation.
Overview #
Pachyderm state is stored in two main places:
- An object-store holding Pachyderm’s data.
- A PostgreSQL instance made up of one or two databases:
pachydermholding Pachyderm’s metadata anddexholding authentication data.
Backing up a Pachyderm cluster involves snapshotting both the object store and the PostgreSQL database(s), in a consistent state, at a given point in time.
Restoring it involves re-populating the database(s) and the object store using those backups, then recreating a Pachyderm cluster.
- Make sure that you have a bucket for backup use, separate from the object store used by your cluster.
- Depending on the reasons behind your cluster recovery, you might choose to use an existing vs. a new instance of PostgreSQL and/or the object store.
Backup A Standalone Enterprise Server #
Backing up and restoring an Enterprise Server is similar to the backing up and restoring of a regular cluster, with three slight variations:
- The name of its Kubernetes deployment is
pach-enterpriseversuspachdin the case of a regular cluster. - The Enterprise Server does not use an Object Store.
- An Enterprise server only requires a
dexdatabase.
Make sure that pachctl and kubectl are pointing to the right cluster. Check your Enterprise Server context: pachctl config get active-enterprise-context, or pachctl config set active-enterprise-context <my-enterprise-context-name> --overwrite to set it.
- Pause the Enterprise Server like you would pause a regular cluster by running
pachctl enterprise pause(Enterprise users), or usingkubectl.
kubectl users:
There is a difference with the pause of a regular cluster. The deployment of the enterprise server is named pach-enterprise; therefore, the first command should be:
kubectl scale deployment pach-enterprise --replicas 0 
There is no need to pause all the Pachyderm clusters registered to the Enterprise Server to backup the enterprise server; however, pausing the Enterprise server will result in your clusters becoming unavailable.
As a reminder, the Enterprise Server does not use any object-store. Therefore, the backup of the Enterprise Server only consists in backing up the database
dex.Resume the operations on your Enterprise Server by running
pachctl enterprise unpause(Enterprise users) to scale thepach-enterprisedeployment back up. Alternatively, if you usedkubectl, run:kubectl scale deployment pach-enterprise --replicas 1
Restore An Enterprise Server #
Follow the cluster restoration steps while skipping all tasks related to creating and populating a new object-store.
Once your cluster is up and running, check that all your clusters are automatically registered with your new Enterprise Server.