blob-fuse with workload identity with version 1.23.3 not store file in azure storage account

[Xaviou1003]

I try to use csi driver with aks and workload identity.

I install last version 1.23.3 of driver and follow the different steps describe in this documentation -> option#1: static provision with PV.
When the volume is mount, I have this warning :
Skip NodeStageVolume for volume(rg#xxxxxx#xxxx) since clientID xxxxxxxx-xxxx-xxxx-xxxxxxxxx is provided but service account token is empty.

I don't know if this log message could be have an impact on the driver but at the end, volume is mount but azure storage account is not used (no files appears).
And all file that are created in /mnt/blob is store on the node file system.

Could you give me some help please?

Thanks

What you expected to happen:
I expected that files store in the csi volume will be appear in azure storage account

Anything else we need to know?:

Environment:

• CSI Driver version: 1.23.3
• Kubernetes version (use kubectl version):
  Client Version: v1.28.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.3

[andyzhangx]

/kind bug

[andyzhangx]

@Xaviou1003 have you specified serviceAccountName in your pod spec? if yes, could you paste all spec you have set?

[Xaviou1003]

@andyzhangx Thanks for your answer.

Yes I provided serviceAccountName in my pod dspec.

You can find my spec below (I just mask my client id):

---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  name: csi-blob
  namespace: cloudplatform-backend
  annotations: 
    azure.workload.identity/client-id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  labels:
    azure.workload.identity/use: "true"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: blob.csi.azure.com
  name: pv-blob
  namespace: cloudplatform-backend
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: blob-fuse
  mountOptions:
    - -o allow_other
    - --file-cache-timeout-in-seconds=120
  csi:
    driver: blob.csi.azure.com
    # make sure volumeid is unique for every storage blob container in the cluster
    # the # character is reserved for internal use, the / character is not allowed
    volumeHandle: rg#stgmakspocvolumetst#pvctest
    volumeAttributes:
      storageAccount: stgmakspocvolumetst
      containerName: pvctest  # required
      resourceGroup: rg-aks-tst # optional, specified when the storage account is not under AKS node resource group(which is prefixed with "MC_")
      clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # required
      # tenantID: $IDENTITY_TENANT  #optional, only specified when workload identity and AKS cluster are in different tenant
      # subscriptionid: $SUBSCRIPTION #optional, only specified when workload identity and AKS cluster are in different subscription


---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-blob
  namespace: cloudplatform-backend
  labels:
    app: nginx
    azure.workload.identity/use: "true"
spec:
  serviceName: statefulset-blob
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: csi-blob  #required, Pod does not use this service account has no permission to mount the volume
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: statefulset-blob
          image: artifactory-tst.tkub-tst3d5.az.groupemutuel.ch/docker/ops/alpine-gm:3.17
          lifecycle: 
            preStop: 
              exec: 
                command: 
                  - "/bin/sh"
                  - "-c"
                  - rm /mnt/blob/*
          command:
            - "/bin/sh"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/blob/outfile; sleep 1; done
          volumeMounts:
            - name: pv-blob
              mountPath: /mnt/blob
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: pv-blob
      spec:
        storageClassName: blob-fuse
        accessModes: ["ReadWriteMany"]
        resources:
          requests:
            storage: 10Gi

[Xaviou1003]

Hello @andyzhangx,

Have you had time to look at this problem?

[andyzhangx]

/assign @cvvz

[cvvz]

@Xaviou1003 Could you share the full logs of csi driver? The log message Skip NodeStageVolume for volume(rg#xxxxxx#xxxx) since clientID xxxxxxxx-xxxx-xxxx-xxxxxxxxx is provided but service account token is empty. is normal and we need more logs to investigate what exactly happened.

[Xaviou1003]

@cvvz yes, of course.

The full log of csi-blob-node:

utils.go:110] GRPC response: {}
nodeserver.go:156] NodePublishVolume: volume rg#stgmakspocvolumetst#pvctest mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount at /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount successfully
mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount)
mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount)
nodeserver.go:140] NodePublishVolume: volume rg#stgmakspocvolumetst#pvctest mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount at /var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount with mountOptions: [bind]
utils.go:104] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount","target_path":"/var/lib/kubelet/pods/4f0c9e90-26a9-4033-83ce-301a30fdea0e/volumes/kubernetes.io~csi/pv-blob/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120"]}},"access_mode":{"mode":5}},"volume_context":{"clientID":"xxxxxxxx-xxxx-xxxx-xxxxxxxxx","containerName":"pvctest","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"statefulset-blob-0","csi.storage.k8s.io/pod.namespace":"cloudplatform-backend","csi.storage.k8s.io/pod.uid":"4f0c9e90-26a9-4033-83ce-301a30fdea0e","csi.storage.k8s.io/serviceAccount.name":"csi-blob","resourceGroup":"rg-aks-tst","storageAccount":"stgmakspocvolumetst"},"volume_id":"rg#stgmakspocvolumetst#pvctest"}
utils.go:103] GRPC call: /csi.v1.Node/NodePublishVolume
utils.go:110] GRPC response: {}
nodeserver.go:256] Skip NodeStageVolume for volume(rg#stgmakspocvolumetst#pvctest) since clientID xxxxxxxx-xxxx-xxxx-xxxxxxxxx is provided but service account token is empty
utils.go:104] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120"]}},"access_mode":{"mode":5}},"volume_context":{"clientID":"xxxxxxxx-xxxx-xxxx-xxxxxxxxx","containerName":"pvctest","resourceGroup":"rg-aks-tst","storageAccount":"stgmakspocvolumetst"},"volume_id":"rg#stgmakspocvolumetst#pvctest"}
utils.go:103] GRPC call: /csi.v1.Node/NodeStageVolume

[cvvz]

Thanks, could you also get the kubelet logs?

[cvvz]

And please run kubectl get csidriver and paste the output

[Xaviou1003]

kubectl get csidriver
NAME                 ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES                  AGE
blob.csi.azure.com   false            true             false             <unset>         false               Persistent,Ephemeral   13d
csi.tigera.io        true             true             false             <unset>         false               Ephemeral              289d
disk.csi.azure.com   true             false            false             <unset>         false               Persistent             448d

kubelet log :

reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
reconciler_common.go:228] "operationExecutor.MountVolume started for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:662] "MountVolume.MountDevice succeeded for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/e671026d6647fcedc52e6b5b45dc122d700ded97a38d615a131eabf8cff69883/globalmount\"" pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:744] "MountVolume.SetUp succeeded for volume \"pv-blob\" (UniqueName: \"kubernetes.io/csi/blob.csi.azure.com^rg#stgmakspocvolumetst#pvctest\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:744] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2mq\" (UniqueName: \"kubernetes.io/projected/f5e37abd-3eae-4e9e-a03c-af11854f1306-kube-api-access-hk2mq\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
operation_generator.go:744] "MountVolume.SetUp succeeded for volume \"azure-identity-token\" (UniqueName: \"kubernetes.io/projected/f5e37abd-3eae-4e9e-a03c-af11854f1306-azure-identity-token\") pod \"statefulset-blob-0\" (UID: \"f5e37abd-3eae-4e9e-a03c-af11854f1306\") " pod="cloudplatform-backend/statefulset-blob-0"
util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cloudplatform-backend/statefulset-blob-0"
kubelet.go:2231] "SyncLoop (PLEG): event for pod" pod="cloudplatform-backend/statefulset-blob-0" event=&{ID:f5e37abd-3eae-4e9e-a03c-af11854f1306 Type:ContainerStarted Data:49bb4893c7da67ed2da1ce317818f74bc2bbfcfeaf957f941e55f13649912f69}
kubelet.go:2231] "SyncLoop (PLEG): event for pod" pod="cloudplatform-backend/statefulset-blob-0" event=&{ID:f5e37abd-3eae-4e9e-a03c-af11854f1306 Type:ContainerStarted Data:2244679131653363f75aa491b9e14296fa42ade4e8e053769d1906fb7305e5f2}

[cvvz]

You didn't install blob csi driver correctly. If the csi driver was installed correctly, then the TOKENREQUESTS field should be api://AzureADTokenExchange, it was unset currently.

kubectl get csidriver
NAME                 ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   **TOKENREQUESTS**   REQUIRESREPUBLISH   MODES                  AGE
blob.csi.azure.com   false            true             false             **<unset>**         false               Persistent,Ephemeral   13d

You can uninstall the existing blob csi driver first and use below steps to re-install blob csi driver and try again:

helm repo add blob-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/charts
helm repo update blob-csi-driver

helm install blob-csi-driver blob-csi-driver/blob-csi-driver \
--set node.enableBlobfuseProxy=true \
--set node.blobfuseProxy.installBlobfuse=false \
--set node.blobfuseProxy.installBlobfuse2=false \
--namespace kube-system \
--version v1.23.3

[cvvz]

@Xaviou1003 Currently AKS managed blob csi driver doesn't support this feature yet, so, if you want to try this feature, you have to disable the managed blob csi driver by running az aks update --disable-blob-driver first and then install the open-source blob csi driver manually by following the steps above. We will support this feature in managed blob csi driver from AKS v1.29.

[andyzhangx]

@Xaviou1003 if you have installed open source driver, then you could use following command to update blob.csi.azure.com csidriver object, and then retry should work.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/csi-blob-driver.yaml