Tolerate the node.kubernetes.io/not-ready:NoSchedule by default.

We need to run the CCM to cold start a new cluster; without it, nodes won't be
assigned providerIDs. However, we also can't schedule the CCM pod onto a new
cluster whose nodes set the `node.kubernetes.io/not-ready:NoSchedule` taint.
This is a circular dependency: nodes can't become ready until the CCM assigns
them a providerID, but the CCM can't run until the node is un-tainted. This
patch configures the CCM to tolerate the not-ready taint by default.

Author: jmcarp

charts/oxide-cloud-controller-manager/templates/deployment.yaml

@@ -27,6 +27,9 @@ spec:
           operator: "Exists"
         - key: "node-role.kubernetes.io/control-plane"
           effect: NoSchedule
+        - key: node.kubernetes.io/not-ready
+          operator: Exists
+          effect: "NoSchedule"
       containers:
       - name: {{ include "oxide-ccm.name" . }}
         image: "{{ .Values.deployment.image.name }}:{{ .Values.deployment.image.tag | default .Chart.AppVersion }}"

manifests/oxide-cloud-controller-manager.yaml

@@ -165,6 +165,9 @@ spec:
           operator: "Exists"
         - key: "node-role.kubernetes.io/control-plane"
           effect: NoSchedule
+        - key: node.kubernetes.io/not-ready
+          operator: Exists
+          effect: "NoSchedule"
       containers:
       - name: oxide-cloud-controller-manager
         image: "ghcr.io/oxidecomputer/oxide-cloud-controller-manager:0.6.0"