Skip to content

additional-configuration.adoc

Latest commit

 

History

History
383 lines (325 loc) · 18.8 KB

additional-configuration.adoc

File metadata and controls

383 lines (325 loc) · 18.8 KB

Additional configuration for DevWorkspaces

Table of Contents

How DevWorkspaces are configured

DevWorkspaces allow for additional configuration through the use Devfile attributes and Kubernetes labels and annotations. There are three fields in the DevWorkspace that are used to configure functionality:

  • Devfile attributes defined within the DevWorkspace spec, either at the top-level (.spec.template.attributes) or on individual components. For example:

    kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
spec:
  template:
    attributes:
      my-attribute: my-attribute-value

  • Kubernetes metadata labels (.metadata.labels):

    kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
  labels:
    my-label: my-label-value

  • Kubernetes metadata annotations (.metadata.annotations):

    kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
  labels:
    my-annotation: my-annotation-value

Restricting access to DevWorkspaces

Applying the

controller.devfile.io/restricted-access: "true"

annotation to a DevWorkspace enables additional access control for the workspace. When this annotation is applied to a DevWorkspace:

  • Only the user that created the DevWorkspace can access a terminal in the workspace via pods/exec

  • Only the DevWorkspace Operator serviceaccount or the user that created the DevWorkspace can modify fields in the DevWorkspace custom resource.

This is useful in case a DevWorkspace is expected to contain sensitive information.

Configuring persistent storage used for a DevWorkspace

The top-level Devfile attribute controller.devfile.io/storage-type can be used to configure persistent storage for DevWorkspaces:

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
spec:
  template:
    attributes:
      controller.devfile.io/storage-type: <storage-type>

where <storage-type> is one of

  • per-user: Use one PVC for all workspace volumes, mounting Devfile volumes in subpaths within the shared PVC

  • common: An alias of the per-user storage-type, which behaves the same way as the per-user storage-type. Exists for legacy compatibility reasons.

  • per-workspace: Every workspace is given its own PVC. Each Devfile volume is mounted as a subpath within the workspace PVC.

  • ephemeral: Replace all volumes with emptyDir volumes. This storage type is non-persistent; any local changes will be lost when the workspace is stopped. This is the equivalent of marking all volumes in the Devfile as ephemeral: true

  • async: Use emptyDir volumes for workspace volumes, but include a sidecar that synchronises local changes to a persistent volume as in the common strategy. This can potentially avoid issues where mounting volumes to a workspace on startup takes a long time.

Configuring project cloning

The top-level Devfile attribute controller.devfile.io/project-clone can be used to configure how storage is mounted to workspaces. By default, the DevWorkspace Operator will add an init container to the workspace deployment that will clone any projects to the workspace before start. This can be disabled by setting controller.devfile.io/project-clone: disable in the attributes field:

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
spec:
  template:
    attributes:
      controller.devfile.io/project-clone: disable

Configuring sparse checkout for projects

The project-level attribute sparseCheckout can be used to enable a sparse checkout for a given project. The value of this attribute should be a list of paths within the project that should be included in the sparse checkout, separated by spaces. For example, the project

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
spec:
  template:
    projects:
      - name: devworkspace-operator
        attributes:
          sparseCheckout: "docs"
        git:
          remotes:
            origin: "https://github.com/devfile/devworkspace-operator.git"

will clone the DevWorkspace Operator sparsely, so only the docs directory is present.

For more information on sparse checkouts, see documentation for [git sparse-checkout](https://git-scm.com/docs/git-sparse-checkout)

Automatically mounting volumes, configmaps, and secrets

Existing configmaps, secrets, and persistent volume claims on the cluster can be configured by applying the appropriate labels. To mark a resource for mounting to workspaces, apply the label

metadata:
  labels:
    controller.devfile.io/mount-to-devworkspace: "true"

to the resource. For secrets and configmaps, it’s also necessary to apply an additional label:

  • controller.devfile.io/watch-configmap must be applied to configmaps to enable the DevWorkspace Operator to find them on the cluster

  • controller.devfile.io/watch-secret must be applied to secrets to enable the DevWorksapce Operator to find them on the cluster.

By default, resources will be mounted based on the resource name:

  • Secrets will be mounted to /etc/secret/<secret-name>

  • Configmaps will be mounted to /etc/config/<configmap-name>

  • Persistent volume claims will be mounted to /tmp/<pvc-name>

Mounting resources can be additionally configured via annotations:

  • controller.devfile.io/mount-path: configure where the resource should be mounted

    For persistent volume claims, controller.devfile.io/mount-path also supports a JSON array to mount multiple subdirectories at different paths:

    annotations:
  controller.devfile.io/mount-path: '[{"path":"/var/logs","subPath":"data/logs"},{"path":"/etc/config","subPath":"data/config"}]'

    Each entry requires a path field (the container mount path) and an optional subPath field (the subdirectory within the PVC). An empty or missing annotation falls back to /tmp/<pvc-name>.

  • controller.devfile.io/mount-access-mode: for secrets and configmaps only, configure file permissions on files mounted from this configmap/secret. Permissions can be specified in decimal (e.g. "511") or octal notation by prefixing with a "0" (e.g. "0777")

  • controller.devfile.io/mount-as: for secrets and configmaps only, configure how the resource should be mounted to the workspace

    • If controller.devfile.io/mount-as: file, the configmap/secret will be mounted as files within the mount path. This is the default behavior.

    • If controller.devfile.io/mount-as: subpath, the keys and values in the configmap/secret will be mounted as files within the mount path using subpath volume mounts.

    • If controller.devfile.io/mount-as: env, the keys and values in the configmap/secret will be mounted as environment variables in all containers in the DevWorkspace.

    When "file" is used, the configmap is mounted as a directory within the workspace, erasing any files/directories already present. When "subpath" is used, each key in the configmap/secret is mounted as a subpath volume mount in the mount path, leaving existing files intact but preventing changes to the secret/configmap from propagating into the workspace without a restart.

  • controller.devfile.io/read-only: for persistent volume claims, mount the resource as read-only

Adding image pull secrets to workspaces

Labelling secrets with controller.devfile.io/devworkspace_pullsecret: true marks a secret as the Docker pull secret for the workspace deployment. This should be applied to secrets with docker config types (kubernetes.io/dockercfg and kubernetes.io/dockerconfigjson)

Note: As for automatically mounting secrets, it is necessary to apply the controller.devfile.io/watch-secret label to image pull secrets

Adding git credentials to a workspace

Labelling secrets with controller.devfile.io/git-credential marks the secret as containing git credentials. All git credential secrets will be merged into a single secret (leaving the original resources intact). The merged credentials secret is mounted to /.git-credentials/credentials. See git documentation for details on the file format for this configuration. For example

kind: Secret
apiVersion: v1
metadata:
  name: git-credentials-secret
  labels:
    controller.devfile.io/git-credential: 'true'
    controller.devfile.io/watch-secret: 'true'
type: Opaque
data:
  credentials: https://{USERNAME}:{PERSONAL_ACCESS_TOKEN}@{GIT_WEBSITE}

Note: As for automatically mounting secrets, it is necessary to apply the controller.devfile.io/watch-secret label to git credentials secrets

This will mount a file /tmp/.git-credentials/credentials in all workspace containers, and construct a git config to use this file as a credentials store.

Configuring DevWorkspaces to use SSH keys for Git operations

Git SSH keys can be configured for DevWorkspaces by mounting secrets to workspaces.

Prerequisites:

  • An SSH keypair, with the public key uploaded to the Git provider, that stores your repository.

    • The steps below assume the following environment variables are set:

      • $SSH_KEY: path on disk to private key for SSH keypair (e.g. ~/.ssh/id_ed25519)

      • $SSH_PUB_KEY: path on disk to public key for SSH keypair (e.g. ~/.ssh/id_ed25519.pub)

      • $PASSPHRASE: SSH keypair passphrase (optional). Note: requires setting config.enableExperimentalFeatures: true in the DevWorkspaceOperatorConfig.

      • $NAMESPACE: namespace where workspaces using the SSH keypair will be started.

Process:

  1. Create a ssh_config file that will be mounted to /etc/ssh/ssh_config in workspaces to configure SSH to use the mounted keys:

    cat <<EOF > /tmp/ssh_config
host *
  IdentityFile /etc/ssh/dwo_ssh_key
  StrictHostKeyChecking = no
EOF

  2. Create a secret in the namespace where DevWorkspaces will be started that stores the SSH keypair and configuration

    kubectl create secret -n "$NAMESPACE" generic git-ssh-key \
  --from-file=dwo_ssh_key="$SSH_KEY" \
  --from-file=dwo_ssh_key.pub="$SSH_PUB_KEY" \
  --from-file=ssh_config=/tmp/ssh_config \
  --from-literal=passphrase="$PASSPHRASE"

    Note: If a passphrase is provided, the DevWorkspace Operator adds a postStart event that starts the SSH agent and adds the passphrase. The DevWorkspace Operator also modifies the ~/.bashrc to configure the SSH_AGENT_PID and SSH_AUTH_SOCK environment variables. If you are automatically mounting your own ~/.bashrc with a ConfigMap (see Automatically mounting volumes, configmaps, and secrets) you must add the following in your ~/.bashrc:

    [ -f $HOME/ssh-environment ] && source $HOME/ssh-environment

    Note: Specifying a passphrase for an SSH key is an experimental feature and is controlled by the DevWorkspaceOperatorConfig’s config.enableExperimentalFeatures field.

  3. Annotate the secret to configure automatic mounting to DevWorkspaces

    kubectl patch secret -n "$NAMESPACE" git-ssh-key --type merge -p \
  '{
    "metadata": {
      "labels": {
        "controller.devfile.io/mount-to-devworkspace": "true",
        "controller.devfile.io/watch-secret": "true"
      },
      "annotations": {
        "controller.devfile.io/mount-path": "/etc/ssh/",
        "controller.devfile.io/mount-as": "subpath"
      }
    }
  }'

    This will mount the files in the git-ssh-key secret to /etc/ssh/, creating files /etc/ssh/dwo_ssh_key, /etc/ssh/dwo_ssh_key.pub and overwrite file /etc/ssh/ssh_config with the file created in step 1.

Setting an alternate configuration for a workspace

It is possible to configure a workspace to use an alternate DevWorkspaceOperatorConfig. In order to do so, the alternate DevWorkspaceOperatorConfig must exist on the cluster, and the controller.devfile.io/devworkspace-config workspace attribute must be set. The controller.devfile.io/devworkspace-config attribute takes two string fields: name and namespace.

  • name: the metadata.name of the alternate DevWorkspaceOperatorConfig.

  • namespace: the metadata.namespace of the alternate DevWorkspaceOperatorConfig.

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
spec:
  template:
    attributes:
      controller.devfile.io/devworkspace-config:
        name: <string>
        namespace: <string>

Note: the alternate DevWorkspaceOperatorConfig will be merged with the default DevWorkspaceOperatorConfig, overriding fields in the default configuration. Fields unset in the overridden configuration will use the global values.

Fine-grained configuration of workspace pods and containers

The attributes pod-overrides and container-overrides can be applied to DevWorkspaces in order to configure fields on the Kubernetes objects that are not normally exposed through the Kubernetes API.

The format for overrides is the same as the object being overridden — a pod template in the case of pod-overrides and a container in the case of container-overrides. The value of the attribute is applied via Kubernetes strategic merge to patch the default configuration according to predefined merge rules. By default, strategic merge patch will update objects in place, leaving fields not specified in the patch unchanged, though this can be configured using patch directives. See strategic merge patch documentation for details.

The value of attributes can be specified as yaml or json. In other words, both

pod-overrides:
  metadata:
    labels:
      my-label: my value

and

pod-overrides: {"metadata": {"labels": {"my-label": "my value"}}}

are valid.

Pod overrides

The pod-overrides attribute can be applied to either the top-level attributes field or the attributes field on an individual container component in order to override fields in the workspace deployment’s Pod spec. The value for this attribute should be specified as a Pod template as used in a Kubernetes Deployment (see kubectl explain deployment.spec.template for details). For example, the pod-overrides field below adds the label my-label: my-value to the workspace pod and configures the pod security context

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: example-devworkspace
spec:
  started: true
  template:
    attributes:
      pod-overrides:
        metadata:
          labels:
            my-label: my value
        spec:
          securityContext:
            runAsUser: 1234

Note that the pod-overrides field does not allow configuring the containers and initContainers fields from the pod spec. In order to configure these elements, use the existing DevWorkspace spec.template.components field.

The DevWorkspace Operator sets the pod spec.volumes field by default for config files, metadata, and credentials. To avoid unexpected behaviour, the spec.volumes field should not be overridden.

In the case that the pod-overrides attribute is defined multiple times in a DevWorkspace, each override will be applied in sequence (with later overrides potentially rewriting previous ones) in the following order:

  1. Any overrides specified on container-type components, in the order they appear in the DevWorkspace

  2. Overrides specified in the top-level attributes field on the DevWorkspace.

Container overrides

The container-overrides attribute can be applied to container-type components in a DevWorkspace to override fields in that individual container. The value for this attribute should be specified as a Kubernetes Container (see kubectl explain pods.spec.containers for details). For example, the container-overrides field below configures resource limit for the nvidia.com/gpu extended cluster resource:

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: example-devworkspace
spec:
  started: true
  template:
    components:
      - name: web-terminal
        attributes:
          container-overrides: {"resources":{"limits":{"nvidia.com/gpu":"1"}}}

Note that the container-overrides field does not allow configuring most fields that are available through the regular DevWorkspace API. In particular, workspaces will fail to start if the container-overrides attribute attempts to override name, image, command, args, ports, or env.

The DevWorkspace Operator sets the volumeMounts by default for config files, metadata, and credentials. To avoid unexpected behaviour, the volumeMounts field should not be overridden.

Debugging a failing workspace

Normally, when a workspace fails to start, the deployment will be scaled down and the workspace will be stopped in a Failed state. This can make it difficult to debug misconfiguration errors, so the annotation controller.devfile.io/debug-start: "true" can be applied to DevWorkspaces to leave resources for failed workspaces on the cluster. This allows viewing logs from workspace containers.

It also enables a specialized debug mode for postStart lifecycle hooks, which are often used for initial setup tasks.

When a postStart command fails: - The container will not immediately crash or restart. It would stay in ContainerCreating phase. - The command failure is trapped, and the container is instead forced to sleep for some seconds as per configured DevWorkspace progressTimeout (by default, 5 minutes).

This trap sleep pause is a critical window that allows developers to connect to the container (e.g., using kubectl exec), inspect the file system, and review logs /tmp/poststart-stderr.txt / /tmp/poststart-stdout.txt to diagnose the exact cause of the postStart failure before the workspace ultimately scales down. This applies to both standard and timeout-wrapped postStart scripts.

Setting RuntimeClass for workspace pods

To run a DevWorkspace with a specific RuntimeClass, the attribute controller.devfile.io/runtime-class can be set on the DevWorkspace with the name of the RuntimeClass to be used. If the specified RuntimeClass does not exist, the workspace will fail to start. For example, to run a DevWorkspace using the kata containers runtime in clusters where this is enabled, the DevWorkspace can be specified:

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: my-workspace
spec:
  template:
    attributes:
      controller.devfile.io/runtime-class: kata

For documentation on Runtime Classes, see https://kubernetes.io/docs/concepts/containers/runtime-class/