Welcome to the Linux Foundation Forum!

Lab 8.3: NFS volume fails to mount

Hello,

I was going to ask for some guidance, but just figured out the solution. I'll post it here in-case anyone encounters the same problem.

I was stuck between step 7 & 8 of lab 8.3.

PV and PVC were created OK, but the pod created in step 6 seems to fail to mount the NFS volume and I was uncertain whether I made a mistake or if there's was another problem.

kubectl get pods:

  1. NAME READY STATUS RESTARTS AGE
  2. nginx-nfs-5f58fd64fd-qsqs8 0/1 ContainerCreating 0 16m

kubectl describe pod nginx-nfs-5f58fd64fd-qsqs8:

  1. Name: nginx-nfs-5f58fd64fd-qsqs8
  2. Namespace: default
  3. Priority: 0
  4. Node: k8s-worker/172.31.46.65
  5. Start Time: Wed, 21 Jul 2021 16:14:36 +0000
  6. Labels: pod-template-hash=5f58fd64fd
  7. run=nginx
  8. Annotations: <none>
  9. Status: Pending
  10. IP:
  11. IPs: <none>
  12. Controlled By: ReplicaSet/nginx-nfs-5f58fd64fd
  13. Containers:
  14. nginx:
  15. Container ID:
  16. Image: nginx
  17. Image ID:
  18. Port: 80/TCP
  19. Host Port: 0/TCP
  20. State: Waiting
  21. Reason: ContainerCreating
  22. Ready: False
  23. Restart Count: 0
  24. Environment: <none>
  25. Mounts:
  26. /opt from nfs-vol (rw)
  27. /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jg5t5 (ro)
  28. Conditions:
  29. Type Status
  30. Initialized True
  31. Ready False
  32. ContainersReady False
  33. PodScheduled True
  34. Volumes:
  35. nfs-vol:
  36. Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
  37. ClaimName: pvc-one
  38. ReadOnly: false
  39. kube-api-access-jg5t5:
  40. Type: Projected (a volume that contains injected data from multiple sources)
  41. TokenExpirationSeconds: 3607
  42. ConfigMapName: kube-root-ca.crt
  43. ConfigMapOptional: <nil>
  44. DownwardAPI: true
  45. QoS Class: BestEffort
  46. Node-Selectors: <none>
  47. Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  48. node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  49. Events:
  50. Type Reason Age From Message
  51. ---- ------ ---- ---- -------
  52. Normal Scheduled 16m default-scheduler Successfully assigned default/nginx-nfs-5f58fd64fd-qsqs8 to k8s-worker
  53. Warning FailedMount 3m4s (x3 over 9m54s) kubelet Unable to attach or mount volumes: unmounted volumes=[nfs-vol], unattached volumes=[kube-api-access-jg5t5 nfs-vol]: timed out waiting for the condition
  54. Warning FailedMount 47s (x4 over 14m) kubelet Unable to attach or mount volumes: unmounted volumes=[nfs-vol], unattached volumes=[nfs-vol kube-api-access-jg5t5]: timed out waiting for the condition
  55. Warning FailedMount 10s (x16 over 16m) kubelet MountVolume.SetUp failed for volume "pvvol-1" : mount failed: exit status 32
  56. Mounting command: mount
  57. Mounting arguments: -t nfs k8scp:/opt/sfw /var/lib/kubelet/pods/c79023bd-4e26-4ffd-b065-a198c8c03303/volumes/kubernetes.io~nfs/pvvol-1
  58. Output: mount: /var/lib/kubelet/pods/c79023bd-4e26-4ffd-b065-a198c8c03303/volumes/kubernetes.io~nfs/pvvol-1: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

Googling the error message suggested failure to install nfs-common, but it was installed on the cp node. While preparing this post I realized the pod is running on the worker node which didn't have it.

Running sudo apt -y install nfs-common there and then recreating the pod resolved the problem.

I see I missed that 8.2 step 5 should've been run on the worker node, so my mistake.

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Thanks for this hint!

  • Hi, I did the whole thing, including the "nfs-common" and I am not making any progress. I am not sure if someone has faced the same problem.

  • Hi @lzambra,

    Please provide the sequence of commands you executed on each node, part of the nfs installation, and their corresponding outputs.

    Regards,
    -Chris

  • ok, info to someone that might face the same issue:

    On previous lab, you need to use the namespace "small", if you try to create the new pvc, this is going to use "pvc-one" as name, and you will have two under the same name. I just removed the pvc-one that belongs to small and recreate the pvc without namespace, also, I noticed that didn't work. But the issue here was that PersistentVolume was not created. So, I create the PersistentVolume (PVol.yaml) and after the "pvc.yaml" worked!

  • Hi, I also encountered this error with pod failing to mount nfs volume. However in my case specific error was "Output: mount.nfs: Failed to resolve server cp: Name or service not known"

    I did not have "cp" defined in /etc/hosts on nodes, just "k8scp" as stated in labs. There are many issues on net where pod can't resolve nfs server hostname even if dns is working fine and the solution was to use IP address of the "cp" node in the PVol.yaml instead of hostname:

    server:1.2.3.4

  • Hi,
    I have followed these steps and yet I encounter this issue in lab 9.3 (previously 8.3):
    kubectl describe pod

    1. Name: sakshi-nginx-nfs-68686f6d59-tljf7
    2. Namespace: default
    3. Priority: 0
    4. Service Account: default
    5. Node: sakshi-k8s-worker-node/134.221.126.56
    6. Start Time: Tue, 13 Aug 2024 13:11:12 +0000
    7. Labels: pod-template-hash=68686f6d59
    8. run=nginx
    9. Annotations: <none>
    10. Status: Pending
    11. IP:
    12. IPs: <none>
    13. Controlled By: ReplicaSet/sakshi-nginx-nfs-68686f6d59
    14. Containers:
    15. nginx:
    16. Container ID:
    17. Image: nginx
    18. Image ID:
    19. Port: 80/TCP
    20. Host Port: 0/TCP
    21. State: Waiting
    22. Reason: ContainerCreating
    23. Ready: False
    24. Restart Count: 0
    25. Environment: <none>
    26. Mounts:
    27. /opt from nfs-vol (rw)
    28. /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mqqj7 (ro)
    29. Conditions:
    30. Type Status
    31. Initialized True
    32. Ready False
    33. ContainersReady False
    34. PodScheduled True
    35. Volumes:
    36. nfs-vol:
    37. Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    38. ClaimName: pvc-one
    39. ReadOnly: false
    40. kube-api-access-mqqj7:
    41. Type: Projected (a volume that contains injected data from multiple sources)
    42. TokenExpirationSeconds: 3607
    43. ConfigMapName: kube-root-ca.crt
    44. ConfigMapOptional: <nil>
    45. DownwardAPI: true
    46. QoS Class: BestEffort
    47. Node-Selectors: <none>
    48. Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
    49. node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    50. Events:
    51. Type Reason Age From Message
    52. ---- ------ ---- ---- -------
    53. Normal Scheduled 10m default-scheduler Successfully assigned default/sakshi-nginx-nfs-68686f6d59-tljf7 to sakshi-k8s-worker-node
    54. Warning FailedMount 97s (x4 over 7m57s) kubelet MountVolume.SetUp failed for volume "pvvol-1" : mount failed: exit status 32
    55. Mounting command: mount
    56. Mounting arguments: -t nfs sakshi-k8s-cp-node:/opt/sfw /var/lib/kubelet/pods/a15e58a8-2031-4658-9a31-133257090160/volumes/kubernetes.io~nfs/pvvol-1
    57. Output: mount.nfs: Connection timed out

    This is how my PVol.yaml looks like:

    1. apiVersion: v1
    2. kind: PersistentVolume
    3. metadata:
    4. name: pvvol-1
    5. spec:
    6. capacity:
    7. storage: 1Gi
    8. accessModes:
    9. - ReadWriteMany
    10. persistentVolumeReclaimPolicy: Retain
    11. nfs:
    12. path: /opt/sfw
    13. server: sakshi-k8s-cp-node #<-- Edit to match cp node
    14. readOnly: false
    15.  

    Can anyone suggest a solution or someone else faced something similar?

  • Posts: 2,443

    Hi @sakshi1120,

    Typically completing steps 1 through 4 of lab exercise 9.2 (Creating a Persistent NFS Volume) on the control plane node, followed by step 5 of lab exercise 9.2 on the worker node are sufficient to initiate the NFS server on the control plane node and the NFS client on the worker node respectively. Ensure that the steps work with the k8scp alias as presented, and/or with the control plane node hostname.

    However, there may be network settings at play to impact the desired operations and eventually outcomes in the subsequent lab exercise 9.3. In this particular scenario, can the worker node resolve the control plane node hostname? If in doubt, you can add the control plane node hostname alongside the k8scp recommended alias to the /etc/hosts file of the worker node. The updated entry should look like this cp-private-IP k8scp cp-node-hostname (where you substitute cp-private-IP with the private IP of your control plane node, and cp-node-hostname with the hostname of your control plane node).

    Try to aim for consistency; if the k8scp alias works in lab exercise 9.2 (step 5), then use the same k8scp alias in the PV definition manifest in step 6 of lab exercise 9.2, otherwise, attempt the same with the control plane node hostname instead. I would recommend using the hostname approach because the solution remains operational even in chapter 16 when the k8scp alias will be assigned to another server.

    Regards,
    -Chris

  • Thanks Chris! This worked! I used control plane hostname in the PV definition manifest which the worker node couldn't resolve. Using the k8scp alias solved it.

  • Thank you very much.

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training