Goal:
This article explains how to use nodeSelector to constrain POD csi-controller-kdf-0 to only be able to run on particular Node(s).Env:
MapR 6.1 (secured)MapR CSI 1.0.0
Kubernetes Cluster in GKE
Use case:
For MapR CSI, we want the POD from StatefulSet "csi-controller-kdf" to only run on specific node(s).Solution:
1. List current nodes from Kubernetes cluster
$ kubectl get nodes NAME STATUS ROLES AGE VERSION gke-standard-cluster-1-default-pool-f6e6e4c1-45ql Ready <none> 22m v1.13.11-gke.14 gke-standard-cluster-1-default-pool-f6e6e4c1-fbhp Ready <none> 22m v1.13.11-gke.14 gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 Ready <none> 22m v1.13.11-gke.14 gke-standard-cluster-1-default-pool-f6e6e4c1-r20n Ready <none> 22m v1.13.11-gke.14 gke-standard-cluster-1-default-pool-f6e6e4c1-xr3s Ready <none> 22m v1.13.11-gke.14
For example, we want the POD from StatefulSet "csi-controller-kdf" to only run on node "gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5".
2. Attach a label to this node
kubectl label nodes gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 for-csi-controller=trueHere the label key is "for-csi-controller" and the label value is "true".
Verify that the label is attached on that node:
$ kubectl get nodes -l for-csi-controller=true NAME STATUS ROLES AGE VERSION gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 Ready <none> 34m v1.13.11-gke.14
3. Modify csi-maprkdf-v1.0.0.yaml
cp csi-maprkdf-v1.0.0.yaml csi-maprkdf-v1.0.0_modified.yaml vi csi-maprkdf-v1.0.0_modified.yamlAdd below to the bottom of the definiton for StatefulSet "csi-controller-kdf"
nodeSelector:
for-csi-controller: "true"
One full example for StatefulSet "csi-controller-kdf" is:kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-controller-kdf
namespace: mapr-csi
spec:
serviceName: "kdf-provisioner-svc"
replicas: 1
template:
metadata:
labels:
app: csi-controller-kdf
spec:
serviceAccount: csi-controller-sa
containers:
- name: csi-attacher
image: quay.io/k8scsi/csi-attacher:v1.0.1
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/csi/sockets/pluginproxy/
- name: csi-provisioner
image: quay.io/k8scsi/csi-provisioner:v1.0.1
args:
- "--provisioner=com.mapr.csi-kdf"
- "--csi-address=$(ADDRESS)"
- "--volume-name-prefix=mapr-pv"
- "--v=5"
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/csi/sockets/pluginproxy/
- name: csi-snapshotter
image: quay.io/k8scsi/csi-snapshotter:v1.0.1
imagePullPolicy: "Always"
args:
- "--snapshotter=com.mapr.csi-kdf"
- "--csi-address=$(ADDRESS)"
- "--snapshot-name-prefix=mapr-snapshot"
- "--v=5"
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
volumeMounts:
- name: socket-dir
mountPath: /var/lib/csi/sockets/pluginproxy/
- name: liveness-probe
image: quay.io/k8scsi/livenessprobe:v1.0.1
imagePullPolicy: "Always"
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--connection-timeout=60s"
- "--health-port=9809"
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
volumeMounts:
- name: socket-dir
mountPath: /var/lib/csi/sockets/pluginproxy/
- name: mapr-kdfprovisioner
image: maprtech/csi-kdfprovisioner:1.0.0
imagePullPolicy: "Always"
args :
- "--nodeid=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
- "-v=5"
env:
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://plugin/csi.sock
ports:
- containerPort: 9809
name: healthz
protocol: TCP
livenessProbe:
failureThreshold: 20
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 5
volumeMounts:
- name: socket-dir
mountPath: /plugin
- name: k8s-log-dir
mountPath: /var/log/csi-maprkdf
- name: timezone
mountPath: /etc/localtime
readOnly: true
volumes:
- name: socket-dir
emptyDir: {}
- name: k8s-log-dir
hostPath:
path: /var/log/csi-maprkdf
type: DirectoryOrCreate
- name: timezone
hostPath:
path: /etc/localtime
nodeSelector:
for-csi-controller: "true"
4. Create StatefulSet "csi-controller-kdf" using the modified version when configuring MapR CSI
kubectl apply -f csi-maprkdf-v1.0.0_modified.yamlOther steps to configure MapR CSI are the same as this blog.
5. Verify that POD "csi-controller-kdf-0" is running on that specific node
$ kubectl get pods -n mapr-csi -o wide |grep csi-controller-kdf-0 csi-controller-kdf-0 5/5 Running 0 56m xx.xx.xx.4 gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 <none> <none>
Disaster Recovery Test:
1. Drain this specific node and evict all the PODs except those for DaemonSets.
$ kubectl drain gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 --ignore-daemonsets --delete-local-data node/gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-hzrq7, kube-system/prometheus-to-sd-jxhrm, mapr-csi/csi-nodeplugin-kdf-ssbxp evicting pod "csi-controller-kdf-0" evicting pod "kube-dns-79868f54c5-rggws" pod/csi-controller-kdf-0 evicted pod/kube-dns-79868f54c5-rggws evicted node/gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 evicted
2. Check if the POD "csi-controller-kdf-0" will be rescheduled on other nodes or not.
$ kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ... mapr-csi csi-controller-kdf-0 0/5 Pending 0 16m <none> <none> <none> <none> ...As we can see, the POD "csi-controller-kdf-0" will be pending and can not be rescheduled on other nodes.
This proves that the label is working.
3. Mark the specific node available again
kubectl uncordon gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5
4. Verify that POD "csi-controller-kdf-0" is running on the specific node again
$ kubectl get pods --all-namespaces -o wide |grep -i csi-controller-kdf-0 mapr-csi csi-controller-kdf-0 5/5 Running 0 17m xx.xx.xx.5 gke-standard-cluster-1-default-pool-f6e6e4c1-hzh5 <none> <none>
5. Verify the mount point is working in the test POD
$ kubectl exec -ti testpod -n testns -- ls -altr /mapr total 6 drwxrwxrwt 3 5000 5000 1 Nov 25 11:17 kafka-streams drwxrwxrwt 3 5000 5000 1 Nov 25 11:18 ksql drwxrwxrwx 3 5000 5000 2 Dec 6 12:38 spark drwxr-xr-x 1 root root 4096 Dec 12 22:11 .. drwxr-xr-x 5 5000 5000 3 Dec 12 23:45 .
No comments:
Post a Comment