Environment Variables

You can define environment variables that handle required configuration. In Pachyderm, you can define the following types of environment variables:

• pachd variables: Used for your Pachyderm daemon container.

• Pachyderm worker variables: Used by the Kubernetes pods that run your pipeline code.

💡

You can reference environment variables in your code. For example, if your code writes data to an external system and you want to know the current job ID, you can use the PACH_JOB_ID environment variable to refer to the current job ID.

pachd Environment Variables #

You can find the list of pachd environment variables in the pachd manifest by running the following command:

kubectl get deploy pachd -o yaml

The following tables list all the pachd environment variables.

Global Configuration #

PachD Configuration #

Storage Configuration #

Pipeline Worker Environment Variables #

Pachyderm defines many environment variables for each Pachyderm worker that runs your pipeline code. You can print the list of environment variables into your Pachyderm logs by including the env command into your pipeline specification. For example, if you have an images repository, you can configure your pipeline specification like this:

{
    "pipeline": {
        "name": "env"
    },
    "input": {
        "pfs": {
            "glob": "/",
            "repo": "images"
        }
    },
    "transform": {
        "cmd": ["sh" ],
        "stdin": ["env"],
        "image": "ubuntu:14.04"
    }
}

Run this pipeline and, upon completion, you can view the log with variables by running the following command:

pachctl logs --pipeline=env
PPS_WORKER_IP=172.17.0.7
DASH_PORT_8081_TCP_PROTO=tcp
PACHD_PORT_600_TCP_PORT=600
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
...

You should see a lengthy list of variables. Many of them define internal networking parameters that most probably you will not need to use.

Most users find the following environment variables particularly useful:

In addition to these environment variables, Kubernetes injects others for Services that run inside the cluster. These variables enable you to connect to those outside services, which can be powerful but might also result in processing being retried multiple times.

For example, if your code writes a row to a database, that row might be written multiple times because of retries. Interaction with outside services must be idempotent to prevent unexpected behavior. Furthermore, one of the running services that your code can connect to is Pachyderm itself. This is generally not recommended as very little of the Pachyderm API is idempotent, but in some specific cases it can be a viable approach.