Scale: register a network device(register_netdev) takes long time and high CPU usage
we have a network application in which we make netlink call and in the kernel-module we call register_netdev to create a network device we need. This works without any issues for small numbers.
When we create 8k network devices all at once, I could see it takes 2 minutes approx to complete, after which CPU usage continues to be high, mainly with many daemons named systemd-udevd and ifquery
root@NetwAA:~# pgrep -l systemd
1925 systemd-journal
1946 systemd-udevd
2130 systemd-logind
13381 systemd-udevd
13385 systemd-udevd
13393 systemd-udevd
13399 systemd-udevd
13405 systemd-udevd
13411 systemd-udevd
13415 systemd-udevd
13423 systemd-udevd
13428 systemd-udevd
13432 systemd-udevd
13441 systemd-udevd
13446 systemd-udevd
13452 systemd-udevd
13459 systemd-udevd
13462 systemd-udevd
13468 systemd-udevd
13475 systemd-udevd
13483 systemd-udevd
...
root@NetwAA:~# pgrep -l ifquery
13388 ifquery
13392 ifquery
13397 ifquery
13404 ifquery
13409 ifquery
13417 ifquery
...
top - 00:55:05 up 1:14, 2 users, load average: 70.09, 30.47, 12.17
Tasks: 462 total, 12 running, 450 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19.0 us, 74.1 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15937.0 total, 13024.0 free, 2283.3 used, 629.7 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 13308.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1946 root 20 0 26412 11004 3896 R 99.3 0.1 2:22.81 systemd-u+
13551 root 20 0 7660 6820 1556 D 36.5 0.0 0:12.04 ifquery
13564 root 20 0 5812 4816 1404 R 15.6 0.0 0:10.69 ifquery
13656 root 20 0 2908 1968 1460 D 12.0 0.0 0:10.28 ifquery
13590 root 20 0 4096 3396 1564 D 11.6 0.0 0:10.29 ifquery
13711 root 20 0 3568 2856 1556 D 11.3 0.0 0:10.49 ifquery
13572 root 20 0 3964 3104 1540 D 11.0 0.0 0:12.23 ifquery
13784 root 20 0 4096 3236 1404 R 11.0 0.0 0:10.94 ifquery
13471 root 20 0 2380 1812 1564 D 10.6 0.0 0:10.91 ifquery
13640 root 20 0 8848 8020 1440 D 10.6 0.0 0:10.65 ifquery
13701 root 20 0 8056 7300 1512 D 10.0 0.0 0:11.42 ifquery
13630 root 20 0 3304 2592 1556 D 9.6 0.0 0:11.44 ifquery
13704 root 20 0 4492 3640 1548 D 9.6 0.0 0:10.96 ifquery
13392 root 20 0 9112 8304 1460 D 9.0 0.1 0:11.99 ifquery
13521 root 20 0 7660 6704 1440 D 9.0 0.0 0:12.84 ifquery
13770 root 20 0 8716 7768 1452 D 9.0 0.0 0:12.06 ifquery
High CPU usage by systemd-udevd and ifquery
after a long time: //about 10-15 mins
root@NetwAA:~# udevadm monitor
UDEV [538.005910] add /devices/virtual/net/xe9.924/queues/tx-0 (queues)
UDEV [538.011380] add /devices/virtual/net/xe9.934/queues/rx-0 (queues)
UDEV [538.248471] add /devices/virtual/net/xe9.933 (net)
UDEV [538.266858] add /devices/virtual/net/xe9.976 (net)
UDEV [538.358951] add /devices/virtual/net/xe9.933/queues/rx-0 (queues)
UDEV [538.359507] add /devices/virtual/net/xe9.933/queues/tx-0 (queues)
UDEV [538.359594] add /devices/virtual/net/xe9.936/queues/rx-0 (queues)
UDEV [538.360094] add /devices/virtual/net/xe9.934/queues/tx-0 (queues)
UDEV [538.360121] add /devices/virtual/net/xe9.936/queues/tx-0 (queues)
UDEV [538.609054] add /devices/virtual/net/xe9.954 (net)
UDEV [538.624835] add /devices/virtual/net/xe9.968 (net)
UDEV [538.797069] add /devices/virtual/net/xe9.932 (net)
UDEV [538.804163] add /devices/virtual/net/xe9.954/queues/rx-0 (queues)
UDEV [538.813290] add /devices/virtual/net/xe9.954/queues/tx-0 (queues)
UDEV [538.813778] add /devices/virtual/net/xe9.968/queues/rx-0 (queues)
UDEV [538.814202] add /devices/virtual/net/xe9.968/queues/tx-0 (queues)
UDEV [538.814664] add /devices/virtual/net/xe9.976/queues/rx-0 (queues)
UDEV [538.815270] add /devices/virtual/net/xe9.976/queues/tx-0 (queues)
UDEV [539.581034] add /devices/virtual/net/xe9.953 (net)
UDEV [539.634047] add /devices/virtual/net/xe9.978 (net)
...
...
UDEV [1277.422066] add /devices/virtual/net/xe9.7999/queues/rx-0 (queues)
UDEV [1277.425117] add /devices/virtual/net/xe9.7987 (net)
UDEV [1277.425660] add /devices/virtual/net/xe9.7987/queues/tx-0 (queues)
UDEV [1277.426005] add /devices/virtual/net/xe9.7987/queues/rx-0 (queues)
UDEV [1277.436997] add /devices/virtual/net/xe9.7996 (net)
UDEV [1277.437450] add /devices/virtual/net/xe9.7996/queues/rx-0 (queues)
UDEV [1277.437550] add /devices/virtual/net/xe9.7996/queues/tx-0 (queues)
UDEV [1277.473557] add /devices/virtual/net/xe9.7997 (net)
UDEV [1277.474560] add /devices/virtual/net/xe9.7997/queues/tx-0 (queues)
UDEV [1277.474615] add /devices/virtual/net/xe9.7997/queues/rx-0 (queues)
root@NetwAA:~#
root@NetwAA:~#
root@NetwAA:~# pgrep -l ifquery
root@NetwAA:~# pgrep -l systemd
1 systemd
1925 systemd-journal
1946 systemd-udevd
2130 systemd-logind
root@NetwAA:~#
what is happening in the background in kernel ?
why is the bottle-neck ? is it due to rtnl semaphore access ?
Categories
- All Categories
- 232 LFX Mentorship
- 232 LFX Mentorship: Linux Kernel
- 812 Linux Foundation IT Professional Programs
- 365 Cloud Engineer IT Professional Program
- 183 Advanced Cloud Engineer IT Professional Program
- 82 DevOps Engineer IT Professional Program
- 151 Cloud Native Developer IT Professional Program
- 138 Express Training Courses & Microlearning
- 138 Express Courses - Discussion Forum
- Microlearning - Discussion Forum
- 6.4K Training Courses
- 48 LFC110 Class Forum - Discontinued
- 71 LFC131 Class Forum
- 44 LFD102 Class Forum
- 228 LFD103 Class Forum
- 19 LFD110 Class Forum
- 42 LFD121 Class Forum
- 18 LFD133 Class Forum
- 8 LFD134 Class Forum
- 18 LFD137 Class Forum
- 71 LFD201 Class Forum
- 5 LFD210 Class Forum
- 5 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 2 LFD233 Class Forum
- 4 LFD237 Class Forum
- 24 LFD254 Class Forum
- 702 LFD259 Class Forum
- 111 LFD272 Class Forum - Discontinued
- 4 LFD272-JP クラス フォーラム
- 13 LFD273 Class Forum
- 186 LFS101 Class Forum
- 1 LFS111 Class Forum
- 3 LFS112 Class Forum
- 3 LFS116 Class Forum
- 7 LFS118 Class Forum
- LFS120 Class Forum
- 9 LFS142 Class Forum
- 8 LFS144 Class Forum
- 4 LFS145 Class Forum
- 3 LFS146 Class Forum
- 2 LFS148 Class Forum
- 15 LFS151 Class Forum
- 4 LFS157 Class Forum
- 45 LFS158 Class Forum
- LFS158-JP クラス フォーラム
- 10 LFS162 Class Forum
- 2 LFS166 Class Forum
- 4 LFS167 Class Forum
- 3 LFS170 Class Forum
- 2 LFS171 Class Forum
- 3 LFS178 Class Forum
- 3 LFS180 Class Forum
- 2 LFS182 Class Forum
- 5 LFS183 Class Forum
- 32 LFS200 Class Forum
- 737 LFS201 Class Forum - Discontinued
- 3 LFS201-JP クラス フォーラム - Discontinued
- 19 LFS203 Class Forum
- 135 LFS207 Class Forum
- 2 LFS207-DE-Klassenforum
- 1 LFS207-JP クラス フォーラム
- 302 LFS211 Class Forum
- 56 LFS216 Class Forum
- 52 LFS241 Class Forum
- 48 LFS242 Class Forum
- 38 LFS243 Class Forum
- 15 LFS244 Class Forum
- 5 LFS245 Class Forum
- LFS246 Class Forum
- LFS248 Class Forum
- 52 LFS250 Class Forum
- 2 LFS250-JP クラス フォーラム
- 1 LFS251 Class Forum
- 156 LFS253 Class Forum
- 1 LFS254 Class Forum
- 1 LFS255 Class Forum
- 9 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.3K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 132 LFS260 Class Forum
- 160 LFS261 Class Forum
- 43 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 24 LFS267 Class Forum
- 25 LFS268 Class Forum
- 32 LFS269 Class Forum
- 6 LFS270 Class Forum
- 202 LFS272 Class Forum - Discontinued
- 2 LFS272-JP クラス フォーラム
- 4 LFS147 Class Forum
- 1 LFS274 Class Forum
- 4 LFS281 Class Forum
- 12 LFW111 Class Forum
- 262 LFW211 Class Forum
- 184 LFW212 Class Forum
- 15 SKF100 Class Forum
- 1 SKF200 Class Forum
- 1 SKF201 Class Forum
- 797 Hardware
- 199 Drivers
- 68 I/O Devices
- 37 Monitors
- 104 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 85 Storage
- 759 Linux Distributions
- 82 Debian
- 67 Fedora
- 17 Linux Mint
- 13 Mageia
- 23 openSUSE
- 148 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 354 Ubuntu
- 470 Linux System Administration
- 39 Cloud Computing
- 71 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 102 System Management
- 47 Web Management
- 69 Mobile Computing
- 18 Android
- 38 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 374 Off Topic
- 115 Introductions
- 174 Small Talk
- 24 Study Material
- 807 Programming and Development
- 304 Kernel Development
- 485 Software Development
- 1.8K Software
- 263 Applications
- 183 Command Line
- 3 Compiling/Installing
- 987 Games
- 317 Installation
- 102 All In Program
- 102 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)