BUG LFS243 - LAB 7.1-11 - If ServiceProfiles after pods are meshed, Linkerd Proxies never reload
Before apply step 11, you must restart the deployment of the booksapp's webapp.
In a nutshell
If ServiceProfiles are created after pods are meshed, newer Linkerd proxies never reload them, so their route tables stay empty and
viz routesshows “-” even though real traffic exists.
Explanation from the web
In the setup, Linkerd is functioning correctly at the service level, but linkerd viz routes shows no route-level metrics due to a known issue in certain newer or edge Linkerd versions.
This bug occurs when ServiceProfiles are applied after the pods have already been injected with the Linkerd proxy.
In these versions, the sidecar proxies fail to reload or synchronize the newly created ServiceProfile, leaving their internal route tables empty.
As a result, although traffic flows normally and service-level metrics are collected, no route-specific metrics are generated or exposed to Prometheus, leading viz routes to display only “–” values. The recommended remediation is to restart the affected deployments after the ServiceProfile exists, ensuring that each proxy correctly loads the route definitions; alternatively, upgrading to a stable Linkerd release resolves the issue permanently.
STEP 11 - without restarting (as from the LAB7.1 book)
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp service/webapp
ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99
GET / webapp - - - - -
GET /authors/{id} webapp - - - - -
GET /books/{id} webapp - - - - -
POST /authors webapp - - - - -
POST /authors/{id}/delete webapp - - - - -
POST /authors/{id}/edit webapp - - - - -
POST /books webapp - - - - -
POST /books/{id}/delete webapp - - - - -
POST /books/{id}/edit webapp - - - - -
[DEFAULT] webapp - - - - -
Checking status (1)
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd -n booksapp viz stat deploy NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TCP_CONN authors 1/1 76.20% 8.0rps 2ms 13ms 19ms 7 books 1/1 81.90% 9.8rps 4ms 51ms 90ms 7 traffic 1/1 100.00% 0.3rps 1ms 2ms 2ms 1 webapp 3/3 81.59% 9.2rps 12ms 79ms 96ms 10
Checking status (2)
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz tap -n booksapp deploy/webapp req id=18:0 proxy=in src=10.244.0.23:36126 dst=10.244.0.25:7000 tls=true :method=POST :authority=webapp:7000 :path=/books req id=18:1 proxy=out src=10.244.0.25:41148 dst=10.244.0.27:7002 tls=true :method=POST :authority=books:7002 :path=/books.json rsp id=18:1 proxy=out src=10.244.0.25:41148 dst=10.244.0.27:7002 tls=true :status=201 latency=12516µs end id=18:1 proxy=out src=10.244.0.25:41148 dst=10.244.0.27:7002 tls=true duration=52µs response-length=0B ...
Restart the deployment to let linkerd s proxies reload
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ kubectl rollout restart deploy -n booksapp deployment.apps/authors restarted deployment.apps/books restarted deployment.apps/traffic restarted deployment.apps/webapp restarted
Re-apply STEP 11, check the metrics
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp service/webapp
ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99
GET / webapp 100.00% 0.2rps 15ms 20ms 20ms
GET /authors/{id} webapp 100.00% 0.1rps 10ms 19ms 20ms
GET /books/{id} webapp 100.00% 0.4rps 11ms 19ms 20ms
POST /authors webapp 100.00% 0.2rps 15ms 20ms 20ms
POST /authors/{id}/delete webapp 100.00% 0.1rps 23ms 29ms 30ms
POST /authors/{id}/edit webapp - - - - -
POST /books webapp 50.00% 0.2rps 17ms 37ms 39ms
POST /books/{id}/delete webapp 100.00% 0.1rps 8ms 17ms 20ms
POST /books/{id}/edit webapp 50.00% 0.2rps 75ms 98ms 100ms
[DEFAULT] webapp - - - - -
Comments
-
Bug in STEP 12
guys, as well step 12 requires modification:
from the web
That
kubectl patchcommand applies a JSON Patch to thewebappDeployment in thebooksappnamespace, adding the annotationlinkerd.io/proxy-external-service: "true"underspec.template.metadata.annotations.
Because the patch modifies the Pod template, Kubernetes automatically rolls out a new ReplicaSet and recreates allwebapppods.
Those new pods include fresh Linkerd proxies that start after thebooksServiceProfile has been created and therefore can correctly load and use it.
In combination with restarting thebooksdeployment, this resolves the route-metrics issue and allowslinkerd viz routes --to service/booksto show valid per-route metrics instead of “-”.Applying Step 12 (from the book)
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp deploy/webapp --to service/books ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 DELETE /books/{id}.json books - - - - - GET /books.json books - - - - - GET /books/{id}.json books - - - - - POST /books.json books - - - - - PUT /books/{id}.json books - - - - - [DEFAULT] books - - - - -Patch to restart with new p
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ kubectl -n booksapp patch deploy/webapp \ --type='json' \ -p '[ { "op": "add", "path": "/spec/template/metadata/annotations/linkerd.io~1proxy-external-service", "value": "true" } ]' deployment.apps/webapp patchedluigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp deploy/webapp --to service/books ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 DELETE /books/{id}.json books - - - - - GET /books.json books - - - - - GET /books/{id}.json books - - - - - POST /books.json books - - - - - PUT /books/{id}.json books - - - - - [DEFAULT] books - - - - -restart anything
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ kubectl -n booksapp rollout restart deploy/books deployment.apps/books restarted
works!
luigi_c_lfs243_gmail_com@cp-lfs243-06-10-25:~$ linkerd viz routes -n booksapp deploy/webapp --to service/books ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 DELETE /books/{id}.json books - - - - - GET /books.json books - - - - - GET /books/{id}.json books 100.00% 0.1rps 4ms 9ms 10ms POST /books.json books 100.00% 0.0rps 15ms 20ms 20ms PUT /books/{id}.json books 0.00% 0.1rps 50ms 95ms 99ms [DEFAULT] books - - - - -0
Categories
- All Categories
- 177 LFX Mentorship
- 177 LFX Mentorship: Linux Kernel
- 766 Linux Foundation IT Professional Programs
- 378 Cloud Engineer IT Professional Program
- 174 Advanced Cloud Engineer IT Professional Program
- 75 DevOps IT Professional Program - Discontinued
- 7 DevOps & GitOps IT Professional Program
- 101 Cloud Native Developer IT Professional Program
- 7.6K Training Courses & Learning Paths
- 4 AI & ML Training
- 1 Blockchain & Decentralized Identity Training
- 10 Cloud & Containers Training
- 1 Cybersecurity Training
- 2 DevOps & Site-Reliability Training
- 1 Linux Kernel Development Training
- 1 Networking Training
- 2 Open Source Best Practice Training
- 2 System Administration Training
- 1 System Engineering Training
- 1 Web & Application Development Training
- 796 Hardware
- 202 Drivers
- 68 I/O Devices
- 37 Monitors
- 95 Multimedia
- 173 Networking
- 91 Printers & Scanners
- 91 Storage
- 770 Linux Distributions
- 81 Debian
- 68 Fedora
- 23 Linux Mint
- 13 Mageia
- 24 openSUSE
- 150 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 356 Ubuntu
- 465 Linux System Administration
- 31 Cloud Computing
- 73 Command Line/Scripting
- Github systems admin projects
- 98 Linux Security
- 78 Network Management
- 101 System Management
- 46 Web Management
- 116 Mobile Computing
- 20 Android
- 81 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 395 Off Topic
- 121 Introductions
- 30 Study Material
- 997 Programming and Development
- 310 Kernel Development
- 669 Software Development
- 1K Software
- 398 Applications
- 182 Command Line
- 5 Compiling/Installing
- 69 Games
- 318 Installation
- Archived
- 183 Small Talk
- 2 LFD140 Class Forum
- 1.4K LFS258 Class Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)