BUG LFS243 - LAB 7.1-11 - If ServiceProfiles after pods are meshed, Linkerd Proxies never reload

luigicucciolillo

Before apply step 11, you must restart the deployment of the booksapp's webapp.

In a nutshell

If ServiceProfiles are created after pods are meshed, newer Linkerd proxies never reload them, so their route tables stay empty and viz routes shows “-” even though real traffic exists.

Explanation from the web

In the setup, Linkerd is functioning correctly at the service level, but linkerd viz routes shows no route-level metrics due to a known issue in certain newer or edge Linkerd versions.
This bug occurs when ServiceProfiles are applied after the pods have already been injected with the Linkerd proxy.
In these versions, the sidecar proxies fail to reload or synchronize the newly created ServiceProfile, leaving their internal route tables empty.
As a result, although traffic flows normally and service-level metrics are collected, no route-specific metrics are generated or exposed to Prometheus, leading viz routes to display only “–” values. The recommended remediation is to restart the affected deployments after the ServiceProfile exists, ensuring that each proxy correctly loads the route definitions; alternatively, upgrading to a stable Linkerd release resolves the issue permanently.

STEP 11 - without restarting (as from the LAB7.1 book)

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp service/webapp
ROUTE                       SERVICE   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
GET /                       webapp      -        -        -              -              -
GET /authors/{id}           webapp      -        -        -              -              -
GET /books/{id}             webapp      -        -        -              -              -
POST /authors               webapp      -        -        -              -              -
POST /authors/{id}/delete   webapp      -        -        -              -              -
POST /authors/{id}/edit     webapp      -        -        -              -              -
POST /books                 webapp      -        -        -              -              -
POST /books/{id}/delete     webapp      -        -        -              -              -
POST /books/{id}/edit       webapp      -        -        -              -              -
[DEFAULT]                   webapp      -        -        -              -              -

Checking status (1)

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd -n booksapp viz stat deploy
NAME      MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
authors      1/1    76.20%   8.0rps           2ms          13ms          19ms          7
books        1/1    81.90%   9.8rps           4ms          51ms          90ms          7
traffic      1/1   100.00%   0.3rps           1ms           2ms           2ms          1
webapp       3/3    81.59%   9.2rps          12ms          79ms          96ms         10

Checking status (2)

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz tap -n booksapp deploy/webapp
req id=18:0 proxy=in  src=10.244.0.23:36126 dst=10.244.0.25:7000 tls=true :method=POST :authority=webapp:7000 :path=/books
req id=18:1 proxy=out src=10.244.0.25:41148 dst=10.244.0.27:7002 tls=true :method=POST :authority=books:7002 :path=/books.json
rsp id=18:1 proxy=out src=10.244.0.25:41148 dst=10.244.0.27:7002 tls=true :status=201 latency=12516µs
end id=18:1 proxy=out src=10.244.0.25:41148 dst=10.244.0.27:7002 tls=true duration=52µs response-length=0B

...

Restart the deployment to let linkerd s proxies reload

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ kubectl rollout restart deploy -n booksapp
deployment.apps/authors restarted
deployment.apps/books restarted
deployment.apps/traffic restarted
deployment.apps/webapp restarted

Re-apply STEP 11, check the metrics

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp service/webapp
ROUTE                       SERVICE   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
GET /                        webapp   100.00%   0.2rps          15ms          20ms          20ms
GET /authors/{id}            webapp   100.00%   0.1rps          10ms          19ms          20ms
GET /books/{id}              webapp   100.00%   0.4rps          11ms          19ms          20ms
POST /authors                webapp   100.00%   0.2rps          15ms          20ms          20ms
POST /authors/{id}/delete    webapp   100.00%   0.1rps          23ms          29ms          30ms
POST /authors/{id}/edit      webapp         -        -             -             -             -
POST /books                  webapp    50.00%   0.2rps          17ms          37ms          39ms
POST /books/{id}/delete      webapp   100.00%   0.1rps           8ms          17ms          20ms
POST /books/{id}/edit        webapp    50.00%   0.2rps          75ms          98ms         100ms
[DEFAULT]                    webapp         -        -             -             -             -

luigicucciolillo

Bug in STEP 12

guys, as well step 12 requires modification:

from the web

That kubectl patch command applies a JSON Patch to the webapp Deployment in the booksapp namespace, adding the annotation linkerd.io/proxy-external-service: "true" under spec.template.metadata.annotations.
Because the patch modifies the Pod template, Kubernetes automatically rolls out a new ReplicaSet and recreates all webapp pods.
Those new pods include fresh Linkerd proxies that start after the books ServiceProfile has been created and therefore can correctly load and use it.
In combination with restarting the books deployment, this resolves the route-metrics issue and allows linkerd viz routes --to service/books to show valid per-route metrics instead of “-”.

Applying Step 12 (from the book)

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp deploy/webapp --to service/books
ROUTE                     SERVICE   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
DELETE /books/{id}.json     books         -     -             -             -             -
GET /books.json             books         -     -             -             -             -
GET /books/{id}.json        books         -     -             -             -             -
POST /books.json            books         -     -             -             -             -
PUT /books/{id}.json        books         -     -             -             -             -
[DEFAULT]                   books         -     -             -             -             -

Patch to restart with new p

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ kubectl -n booksapp patch deploy/webapp \
  --type='json' \
  -p '[ 
    { "op": "add", "path": "/spec/template/metadata/annotations/linkerd.io~1proxy-external-service", "value": "true" }
  ]'
deployment.apps/webapp patched

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp deploy/webapp --to service/books
ROUTE                     SERVICE   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
DELETE /books/{id}.json     books         -     -             -             -             -
GET /books.json             books         -     -             -             -             -
GET /books/{id}.json        books         -     -             -             -             -
POST /books.json            books         -     -             -             -             -
PUT /books/{id}.json        books         -     -             -             -             -
[DEFAULT]                   books         -     -             -             -             -

restart anything

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ kubectl -n booksapp rollout restart deploy/books
deployment.apps/books restarted

works!

luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp deploy/webapp --to service/books
ROUTE                     SERVICE   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
DELETE /books/{id}.json     books         -        -             -             -             -
GET /books.json             books         -        -             -             -             -
GET /books/{id}.json        books   100.00%   0.1rps           4ms           9ms          10ms
POST /books.json            books   100.00%   0.0rps          15ms          20ms          20ms
PUT /books/{id}.json        books     0.00%   0.1rps          50ms          95ms          99ms
[DEFAULT]                   books         -        -             -             -             -