Pachyderm Docs: Global Config

You can set global defaults for your cluster that are passed down to all pipeline specs. These defaults provide a consistent experience for your data scientists and help manage your cluster.

Before You Start #

You must have a running Pachyderm cluster
You should be familiar with the Pachyderm Pipeline Specification (PPS)

Cluster Defaults #

Set Global Cluster Defaults #

Tool:

pachctl update defaults --cluster <<EOF
{
  "createPipelineRequest": {
    "resourceRequests": {
      "cpu": 1,
      "memory": "256Mi",
      "disk": "1Gi"
    },
    "resourceLimits": {
      "cpu": 1,
      "memory": "256Mi",
      "disk": "1Gi"
    }
  }
}
EOF

Log in to Console
Navigate to the Project Dashboard homepage.
Select Cluster Defaults.

Input your desired default Pipeline Specification (PPS) settings in JSON format.

{
 "createPipelineRequest": {
   "resourceRequests": {
     "cpu": 1,
     "memory": "256Mi",
     "disk": "1Gi"
   },
   "sidecarResourceRequests": {
     "cpu": 1,
     "memory": "256Mi",
     "disk": "10Gi"
   }
 }
}

Optionally select the Prettify button to make the JSON more readable.
Select Continue.
Choose from one of the following options:
- Save Cluster Defaults
  - Only new and edited pipelines apply the new defaults
  - No pipeline regeneration or datum reprocessing occurs
- Save Cluster Defaults and Regenerate Pipelines
  - Only pipelines with impacted specs are regenerated
  - Previously processed datums are not reprocessed
- Save Cluster Defaults, Regenerate Pipelines, and Reprocess Datums
  - Only pipelines with impacted specs are regenerated
  - Previously processed datums are reprocessed
Select Save.

ℹ️

Setting global defaults does not trigger existing pipelines to regenerate automatically unless the --regenerate flag is set.

View Global Cluster Defaults #

Open a terminal window.
Input the following command:
```
pachctl inspect defaults --cluster | jq
```

Project Defaults #

Set Project-Level Defaults #

Tool:

pachctl update defaults --project <<EOF
{
  "createPipelineRequest": {
    "resourceRequests": {
      "cpu": 1,
      "memory": "256Mi",
      "disk": "1Gi"
    },
    "resourceLimits": {
      "cpu": 1,
      "memory": "256Mi",
      "disk": "1Gi"
    }
  }
}
EOF

Log in to Console
Navigate to the Project Dashboard homepage.
Scroll to the project you wish to add defaults to and select View Project.
Select Project Defaults from the sidebar.

Input your desired default Pipeline Specification (PPS) settings in JSON format.

{
 "createPipelineRequest": {
   "resourceRequests": {
     "cpu": 1,
     "memory": "256Mi",
     "disk": "1Gi"
   },
   "sidecarResourceRequests": {
     "cpu": 1,
     "memory": "256Mi",
     "disk": "10Gi"
   }
 }
}

Optionally select the Prettify button to make the JSON more readable.
Select Continue.
Choose from one of the following options:
- Save Project Defaults
  - Only new and edited pipelines apply the new defaults
  - No pipeline regeneration or datum reprocessing occurs
- Save Project Defaults and Regenerate Pipelines
  - Only pipelines with impacted specs are regenerated
  - Previously processed datums are not reprocessed
- Save Project Defaults, Regenerate Pipelines, and Reprocess Datums
  - Only pipelines with impacted specs are regenerated
  - Previously processed datums are reprocessed
Select Save.

ℹ️

Setting project-level defaults does not trigger existing pipelines to regenerate automatically unless the --regenerate flag is set.

View Project-Level Defaults #

Open a terminal window.
Input the following command:
```
pachctl inspect defaults --project | jq
```

Override Cluster or Project Defaults #

Users can override global cluster or project defaults in their pipeline specs when necessary. For example, let’s say that you set a global default for resource requests to look like the following:

{
  "resourceRequests": {
    "cpu": 1,
    "memory": "256Mi",
    "disk": "1Gi",
  },
}

A user could disable needing any CPU by setting the following in their pipeline spec:

{
  "resourceRequests": {
    "cpu": null,
  },
}

Any keys not specified in the pipeline spec will apply the global defaults in the following way: project > cluster.