The smb volume can't be mounted to the windows pod after reboot

[willinwu]

I have installed a k8s environment with two linux worker nodes and one windows worker node.

k8s version: 1.20

kubectl get nodes
NAME                              STATUS   ROLES                  AGE   VERSION
docp-k8s-master1                  Ready    control-plane,master   24d   v1.20.1
docp-k8s-worker-11                Ready    worker                 24d   v1.20.1
docp-k8s-worker-21                Ready    worker                 24d   v1.20.1
docp-win191                       Ready    <none>                 23d   v1.20.0

csi-smb-driver version: 0.6.0
kubectl get pod -n kube-system | grep csi

csi-smb-controller-645cd7b449-6dm2d                     3/3     Running   4          21d
csi-smb-controller-645cd7b449-qzr6m                     3/3     Running   7          21d
csi-smb-node-4t5n7                                      3/3     Running   1          21d
csi-smb-node-9j5xm                                      3/3     Running   1          21d
csi-smb-node-kbtb6                                      3/3     Running   1          21d
csi-smb-node-win-mdrv7                                  3/3     Running   3          109m

csi-proxy version: 0.22

I0201 02:13:57.776512    6596 main.go:55] Starting CSI-Proxy Server ...
I0201 02:13:57.781666    6596 main.go:56] Version: v0.2.2-0-gffb169f

sc.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb
provisioner: smb.csi.k8s.io
parameters:
  source: "//docp-smb1/smbservice"
  csi.storage.k8s.io/node-stage-secret-name: "smbcreds"
  csi.storage.k8s.io/node-stage-secret-namespace: "default"
  createSubDir: "true"  # optional: create a sub dir for new volume
reclaimPolicy: Retain  # only retain is supported
volumeBindingMode: Immediate
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=1001
  - gid=1001

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-smb-2
  labels:
    app: busybox-2
spec:
  replicas: 1
  template:
    metadata:
      name: busybox-2
      labels:
        app: busybox-2
    spec:
      tolerations:
        - key: "OS"
          operator: "Equal"
          value: "Windows"
          effect: "NoSchedule"
      nodeSelector:
        "kubernetes.io/os": windows
      containers:
        - name: busybox-2
          image: e2eteam/busybox:1.29
          command:
            - "powershell.exe"
            - "-Command"
            - "while (1) { Add-Content -Encoding Ascii C:\\mnt\\smb\\data.txt $(Get-Date -Format u); sleep 1 }"
          volumeMounts:
            - name: smb
              mountPath: "/mnt/smb"
      volumes:
        - name: smb
          persistentVolumeClaim:
            claimName: pvc-smb-test
  selector:
    matchLabels:
      app: busybox-2

I got the error message like this after reboot the windows node.

MountVolume.MountDevice failed for volume "pvc-1efb71f1-ab8a-4bbf-8db7-84a8e58877b4" : rpc error: code = Internal desc = volume(pvc-1efb71f1-ab8a-4bbf-8db7-84a8e58877b4) mount "//docp-smb1/smbservice" on "\var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-1efb71f1-ab8a-4bbf-8db7-84a8e58877b4\globalmount" failed with smb mapping failed with error: rpc error: code = Unknown desc = NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user \r\nname, are not allowed. Disconnect all previous connections to the server or shared resource and try again. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n + CategoryInfo : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \r\n pping], CimException\r\n + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping\r\n \r\n", err: exit status 1

[andyzhangx]

could you provide the csi-proxy logs? thanks.
csi-proxy code is here, need to check what went wrong:
https://github.com/kubernetes-csi/csi-proxy/blob/v0.2.2/internal/server/smb/server.go#L43-L72

[willinwu]

I can only get log from the csi-procy console like below. Is there any other logs I can provide?

E0201 04:05:06.630924 9220 server.go:69] failed NewSmbGlobalMapping NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user \r\nname, are not allowed. Disconnect all previous connections to the server or shared resource and try again. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n + CategoryInfo : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \r\n pping], CimException\r\n + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping\r\n \r\n", err: exit status 1

[andyzhangx]

there should be a log file, e.g. csi-proxy.log

[willinwu]

It seems there are no other useful information.

Log file created at: 2021/02/01 17:04:05
Running on machine: docp-win191
Binary: Built with gc go1.15.2 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0201 17:04:05.905442   14124 main.go:125] Running as a Windows service.
I0201 17:04:05.908391   14124 main.go:135] Windows Service initialized through SCM
I0201 17:04:05.916432   14124 main.go:55] Starting CSI-Proxy Server ...
I0201 17:04:05.916432   14124 main.go:56] Version: v0.2.2-0-gffb169f
E0201 17:05:22.544719   14124 server.go:69] failed NewSmbGlobalMapping NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user \r\nname, are not allowed. Disconnect all previous connections to the server or shared resource and try again. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \r\n   pping], CimException\r\n    + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping\r\n \r\n", err: exit status 1
E0201 17:07:31.919044   14124 server.go:69] failed NewSmbGlobalMapping NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user \r\nname, are not allowed. Disconnect all previous connections to the server or shared resource and try again. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \r\n   pping], CimException\r\n    + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping\r\n \r\n", err: exit status 1
E0201 17:09:41.261347   14124 server.go:69] failed NewSmbGlobalMapping NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user \r\nname, are not allowed. Disconnect all previous connections to the server or shared resource and try again. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \r\n   pping], CimException\r\n    + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping\r\n \r\n", err: exit status 1
E0201 17:11:50.234415   14124 server.go:69] failed NewSmbGlobalMapping NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user \r\nname, are not allowed. Disconnect all previous connections to the server or shared resource and try again. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \r\n   pping], CimException\r\n    + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping\r\n \r\n", err: exit status 1

[willinwu]

Seems work well after I changed the storageclass source from "//docp-smb1/smbservice" to ip like "//10...*/smbservice".
Help by this link: https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/cannot-connect-to-network-share

Not sure it's the root cause.

[willinwu]

After reboot, reproduced.
Restore the snapshot or change the sc (the remote share path changed) will be useful. But not forever.
And delete the smb driver and create again is no use.
The log is just like before.

Seems like something is cached on the windows worker node.
I just want to know if it can come back by run some command on the windows worker node.

[andyzhangx]

After reboot, reproduced.
Restore the snapshot or change the sc (the remote share path changed) will be useful. But not forever.
And delete the smb driver and create again is no use.
The log is just like before.

Seems like something is cached on the windows worker node.
I just want to know if it can come back by run some command on the windows worker node.

to workaround, run following commands on the agent node:

• Get-SmbGlobalMapping to get the SMB mapping
• Remove-SmbGlobalMapping to remove the existing mapping

[Mohkam]

Would it be safe to remove Smb Global Mapping ? I am afraid this will cause data loss for to the pod already using this mount.

[andyzhangx]

Remove-SmbGlobalMapping won't cause data loss since it just terminate SMB connection, data is stored on remote SMB server.

if agent node already has \\20.xx.xx.56\share connection, it does not allow \\20.xx.xx.56\share\test since it has \\20.xx.xx.56\share, I think it's by design, I could not work out a way to allow both following 2 file share mount works on one node:

• \\20.xx.xx.56\share
• \\20.xx.xx.56\share\test

> Get-SmbGlobalMapping

Status Local Path Remote Path
------ ---------- -----------
OK                \\20.xx.xx.56\share


PS C:\Users\azureuser> New-SmbGlobalMapping -RemotePath \\20.xx.xx.56\share\test -Credential $Credential
New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all
previous connections to the server or shared resource and try again.
At line:1 char:1
+ New-SmbGlobalMapping -RemotePath \\20.xx.xx.56\share\test -Credential ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMapping], CimException
    + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping

Current workaround is only use one SMB file share in one cluster, and createSubDir: "true".

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb
provisioner: smb.csi.k8s.io
parameters:
  source: \\20.xx.xx.56\share
  csi.storage.k8s.io/node-stage-secret-name: "smbcreds"
  csi.storage.k8s.io/node-stage-secret-namespace: "default"
  createSubDir: "true"  # optional: create a sub dir for new volume
reclaimPolicy: Retain  # only retain is supported
volumeBindingMode: Immediate
mountOptions:
  - dir_mode=0777
  - file_mode=0777

[andyzhangx]

Update:
It's more like a Windows bug, after reboot, mount a different smb share will report following error, will reach to windows team for help, stay tuned.

Microsoft Windows [Version 10.0.17763.1697]
(c) 2018 Microsoft Corporation. All rights reserved.

azureuser@2892k8s001 C:\Users\azureuser>powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\azureuser> Get-SmbGlobalMapping

Status Local Path Remote Path
------ ---------- -----------
OK                \\20.xx.xx.56\share\test
OK                \\20.xx.xx.56\share


PS C:\Users\azureuser> $User = "AZURE\USERNAME"
PS C:\Users\azureuser> $PWord = ConvertTo-SecureString -String "PASSWORD" -AsPlainText -Force
PS C:\Users\azureuser> $Credential = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $User, $Pword
PS C:\Users\azureuser> New-SmbGlobalMapping -RemotePath \\20.xx.xx.56\share\test\a -Credential $Credential -Persistent $true
New-SmbGlobalMapping : Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all
previous connections to the server or shared resource and try again.
At line:1 char:1
+ New-SmbGlobalMapping -RemotePath \\20.xx.xx.56\share\test\a -Credenti ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMapping], CimException
    + FullyQualifiedErrorId : Windows System Error 1219,New-SmbGlobalMapping