Welcome to the Linux Foundation Forum!

Lab 11.1 - linkerd-identity pod not running in latest version - known issue?

Posts: 6
in LFS258 Class Forum

Hi, I am trying to get linker running (latest version) on a Centos 7 k8s cluster. The linkerd-identity pod seems to have a problem with corresponding log content of the identity container inside. Did anyone has similar problems?

general info

  1. cat /etc/redhat-release 
  2. CentOS Linux release 7.9.2009 (Core)
  1. kubectl version
  2. Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
  3. Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:09:57Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

procedure

  • linkerd installation went fine
  • linkerd check --pre went fine
  • running linkerd install | apply -f - followed by linkerd check yields:
  1.  Linkerd core checks
  2. ===================
  3.  
  4. kubernetes-api
  5. --------------
  6. √ can initialize the client
  7. √ can query the Kubernetes API
  8.  
  9. kubernetes-version
  10. ------------------
  11. √ is running the minimum Kubernetes API version
  12. √ is running the minimum kubectl version
  13.  
  14. linkerd-existence
  15. -----------------
  16. √ 'linkerd-config' config map exists
  17. √ heartbeat ServiceAccount exist
  18. √ control plane replica sets are ready
  19. √ no unschedulable pods
  20. | No running pods for "linkerd-destination" 
  21.  

observations

  • the linkerd-identity pod remain in status CrashLoopBackOff and all other pods keep waiting
  • container within linkerd-identitypod
  • log from identity container
  • log from linkerd-proxy container

Any help understanding the problem is appreciated.

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Answers

  • Posts: 2,453

    Hi @t.sander,

    Several issues have been reported in the forum with linkerd 2.11. It seems, however, that downgrading to 2.10.0 resolved most of them.

    Regards,
    -Chris

  • Posts: 7

    How do you downgrade? I'll post it if I find it.

  • Posts: 7

    not quite there yet, but i've made some progress:

    export LINKERD2_VERSION=stable-2.10.2
    curl -s -L https://run.linkerd.io/install | sh -
    vi ~/.bashrc

    Add these two lines

    export LINKERD2_VERSION=stable-2.6.0
    export PATH=$PATH:$HOME/.linkerd2/bin

    $ source ~/.bashrc
    $ linkerd version
    Client version: stable-2.6.0
    Server version: unavailable
    $ linkerd check --pre

    ??grant cluser_admin privledge to sa??

    $ linkerd install | kubectl apply -f -

  • Posts: 2,453

    Hi @cccsss000111,

    Downloading the linkerd setup.sh and modifying the linkerd version also works:

    $ curl -sL run.linkerd.io/install > setup.sh
    $ vim setup.sh

    Locate and edit the following line:

    LINKERD2_VERSION=${LINKERD2_VERSION:-stable-2.10.1}

    Then run setup.sh and continue with the rest of the steps from the lab guide:
    $ sh setup.sh

    Regards,
    -Chris

  • Posts: 1

    Hi @chrispokorni, could you also share the process to debug and locate the issue is related to the version? How to cultivate the ideas when debugging this kind of error?

  • Posts: 34
    edited January 2022

    @t.sander
    I am using Virtual Machines and Linkerd Version 11.1 works fine with me.
    Try the logs of the pod.

    kubectl describe pod pod_name -n linkerd

    Check what errors are showing. If still stuck let me know.

  • Posts: 6

    @alihasanahmedk
    Hi, I am running two virtual machines with Centos 7 (as seen in my first post). Exploiting version 2.11.1 I get the following three pods

    1. kubectl -n linkerd get pods                                       
    2.  
    3. NAME                                      READY   STATUS            RESTARTS      AGE
    4. linkerd-destination-75cdb6c9c-nrztn       0/4     PodInitializing   0             2m56s
    5. linkerd-identity-54795b9f9f-5lz6s         0/2     Running           4 (23s ago)   2m56s
    6. linkerd-proxy-injector-6b5699bdcc-75pbx   0/2     PodInitializing   0             2m56s

    The faulty one is the linkerd-identity

    1. kubectl -n linkerd describe pod linkerd-identity-54795b9f9f-5lz6s
    2.  
    3. Events:
    4.   Type     Reason     Age                 From               Message
    5.   ----     ------     ----                ----               -------
    6.   Normal   Scheduled  116s                default-scheduler  Successfully assigned linkerd/linkerd-identity-54795b9f9f-5lz6s to cpu-rsm-cn02
    7.   Normal   Pulled     115s                kubelet            Container image "cr.l5d.io/linkerd/proxy-init:v1.4.0" already present on machine
    8.   Normal   Created    115s                kubelet            Created container linkerd-init
    9.   Normal   Started    115s                kubelet            Started container linkerd-init
    10.   Normal   Created    113s                kubelet            Created container linkerd-proxy
    11.   Normal   Started    113s                kubelet            Started container linkerd-proxy
    12.   Normal   Pulled     113s                kubelet            Container image "cr.l5d.io/linkerd/proxy:stable-2.11.1" already present on machine
    13.   Warning  Unhealthy  85s (x2 over 95s)   kubelet            Liveness probe failed: Get "http://192.168.168.110:9990/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    14.   Warning  Unhealthy  85s (x5 over 111s)  kubelet            Readiness probe failed: Get "http://192.168.168.110:9990/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    15.   Normal   Created    83s (x2 over 113s)  kubelet            Created container identity
    16.   Normal   Pulled     83s (x2 over 113s)  kubelet            Container image "cr.l5d.io/linkerd/controller:stable-2.11.1" already present on machine
    17.   Normal   Started    83s (x2 over 113s)  kubelet            Started container identity
    18.   Warning  Unhealthy  82s (x6 over 111s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

    Regards

  • Posts: 34

    Please share the output of these commands.
    kubectl -n linkerd describe pod linkerd-destination-75cdb6c9c-nrztn
    kubectl -n linkerd describe pod linkerd-proxy-injector-6b5699bdcc-75pbx

  • Posts: 6
    edited January 2022

    These are the complete output of both pods (see attached files)

  • Posts: 34

    According to provided logs, your Pods are working fine. Can you share me the latest result of kubectl get pods -n linkerd

  • Posts: 6 
    1. kubectl -n linkerd get pods 
    2. NAME                                      READY   STATUS             RESTARTS          AGE
    3. linkerd-destination-5df7f655b5-zbk2b      0/4     PodInitializing    0                 19h
    4. linkerd-heartbeat-27393840-2s6pn          0/1     Error              0                 19h
    5. linkerd-heartbeat-27393840-d7n6n          0/1     Error              0                 19h
    6. linkerd-heartbeat-27393840-j848j          0/1     Error              0                 19h
    7. linkerd-heartbeat-27393840-jcgb2          0/1     Error              0                 19h
    8. linkerd-heartbeat-27393840-m8nw4          0/1     Error              0                 19h
    9. linkerd-heartbeat-27393840-s5jjm          0/1     Error              0                 19h
    10. linkerd-heartbeat-27393840-vj6cl          0/1     Error              0                 19h
    11. linkerd-identity-54795b9f9f-p4xfg         0/2     CrashLoopBackOff   367 (4m49s ago)   19h
    12. linkerd-proxy-injector-56b89fc6d4-p98qg   0/2     PodInitializing    0                 19h

    The main problem is shown by the log of the identity container hosted by the linkerd-identity pod

    1. time="2022-02-01T07:49:07Z" level=fatal msg="Failed to initialize identity service: Post \"https://10.96.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": dial tcp 10.96.0.1:443: i/o timeout"
  • Posts: 34

    Try by deleting the identity pod then check the pod again and wait for pod status.
    kubectl delete pod -n linkerd linkerd-identity-54795b9f9f-p4xfg
    Otherwise let me know we will set meet up call to resolve this issue.

  • Posts: 6

    @alihasanahmedk
    As expected the pod is failing again after deleting it:

    1. kubectl -n linkerd get pod
    2. NAME                                      READY   STATUS             RESTARTS      AGE
    3. linkerd-destination-5df7f655b5-zbk2b      0/4     PodInitializing    0             25h
    4. linkerd-heartbeat-27395280-28kl8          0/1     Error              0             66m
    5. linkerd-heartbeat-27395280-659gq          0/1     Error              0             69m
    6. linkerd-heartbeat-27395280-84bqq          0/1     Error              0             77m
    7. linkerd-heartbeat-27395280-bxfmp          0/1     Error              0             72m
    8. linkerd-heartbeat-27395280-dpd75          0/1     Error              0             83m
    9. linkerd-heartbeat-27395280-j9clg          0/1     Error              0             75m
    10. linkerd-heartbeat-27395280-mmlhp          0/1     Error              0             80m
    11. linkerd-identity-54795b9f9f-rd44s         0/2     CrashLoopBackOff   4 (65s ago)   4m
    12. linkerd-proxy-injector-56b89fc6d4-p98qg   0/2     PodInitializing    0             25h
  • Posts: 34
    edited February 2022

    @t.sander we can arrange an online meeting to resolve this issue. you can reach me at alihasanahmedkhan@gmail.com

  • Posts: 2,453

    Hi @t.sander,

    Your describe attachments indicate that your Pods and Nodes may use overlapping subnets. If that is the case, the networking inside your cluster is impacted as a result. Is your Calico using the default 192.168.0.0/16 network? Are your nodes assigned IP addresses from a 192.168.0.0/x subnet?

    Regards,
    -Chris

  • Posts: 6

    Hi @chrispokorni,
    I followed the instructions from the 3.1 Labs and set CALICO_IPV4POOL_CIDR to 192.168.0.0/16 in calico.yaml and edited the kubeadm-config.yaml accordingly.

    The nodes have the following IP adresses
    192.168.149.111/24 (k8scp)
    192.168.149.112/24 (worker)
    so they have the 192.168.149.0/24 network

    The linkerd-identity pod got 192.168.168.109/32

  • Posts: 2,453

    Hi @t.sander,

    As suspected, the two networks overlap, and it eventually causes routing issues within the cluster.

    This can be resolved by ensuring that the two networks do not overlap - either altering the CIDR in calico.yaml and kubead-config.yaml for cluster init, or provisioning the VMs with IP addresses from a different subnet.

    Regards,
    -Chris

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Quick Links

Categories

Upcoming Training