CentOS 7.2 system abnormal restart

zhangdongyu

Hello everyone. Have you ever had any of the following problems？

Our environment is CentOS 7.2. Running Java applications, sometimes the system will restart inexplicably. The screenshot of error reporting is as follows. Please help me.

[BEGIN] 2019/10/19 12:09:34
STATE: TASK_RUNNING (PANIC)
crash> exit
[root@test ~]# crash /usr/lib/debug/lib/modules/3.10.0-327.el7.x86_64/vmlinux /home/vmcore

crash 7.1.5-2.el7
Copyright (C) 2002-2016 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

  KERNEL: /usr/lib/debug/lib/modules/3.10.0-327.el7.x86_64/vmlinux
DUMPFILE: /home/vmcore  [PARTIAL DUMP]
    CPUS: 4
    DATE: Fri Oct 18 14:59:51 2019
  UPTIME: 129 days, 17:06:58

LOAD AVERAGE: 0.00, 0.01, 0.05
TASKS: 1011
NODENAME: XXXXXXXXXXXXXXXX
RELEASE: 3.10.0-327.el7.x86_64
VERSION: #1 SMP Thu Nov 19 22:10:57 UTC 2015
MACHINE: x86_64 (2194 Mhz)
MEMORY: 8 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"
PID: 122524
COMMAND: "java"
TASK: ffff88006e51f300 [THREAD_INFO: ffff8800bb150000]
CPU: 0
STATE: TASK_RUNNING (PANIC)

crash>
crash>
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 1997872 7.6 GB ----
FREE 39523 154.4 MB 1% of TOTAL MEM
USED 1958349 7.5 GB 98% of TOTAL MEM
SHARED 601413 2.3 GB 30% of TOTAL MEM
BUFFERS 0 0 0% of TOTAL MEM
CACHED 718444 2.7 GB 35% of TOTAL MEM
SLAB 54483 212.8 MB 2% of TOTAL MEM

TOTAL SWAP 2097151 8 GB ----
SWAP USED 764 3 MB 0% of TOTAL SWAP
SWAP FREE 2096387 8 GB 99% of TOTAL SWAP

COMMIT LIMIT 3096087 11.8 GB ----
COMMITTED 1466361 5.6 GB 47% of TOTAL LIMIT
crash>
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 ffffffff81951440 RU 0.0 0 0 [swapper/0]

0      0   1  ffff880232590000  RU   0.0       0      0  [swapper/1]
0      0   2  ffff880232590b80  RU   0.0       0      0  [swapper/2]
0      0   3  ffff880232591700  RU   0.0       0      0  [swapper/3]

  1      0   1  ffff880232d78000  IN   0.0  188780   3292  systemd
  2      0   0  ffff880232d78b80  IN   0.0       0      0  [kthreadd]
  3      2   0  ffff880232d79700  IN   0.0       0      0  [ksoftirqd/0]
980      1   0  ffff88022e735080  IN   0.2  689696  23020  rs:main Q:Reg

1157 1 1 ffff88022e333980 IN 0.0 103844 3864 sshd
1164 1 0 ffff880230f0ae00 IN 0.0 24996 408 xinetd
1188 1 0 ffff880230f08b80 IN 0.0 80660 1412 zabbix_agentd
1197 1188 2 ffff88022f1f4500 IN 0.0 80660 1596 zabbix_agentd
1198 1188 3 ffff88022f1f5c00 IN 0.0 80784 2384 zabbix_agentd
1199 1188 3 ffff88022f1f3980 IN 0.0 80784 2384 zabbix_agentd
1200 1188 3 ffff880231fdf300 IN 0.0 80784 2384 zabbix_agentd
1201 1188 0 ffff880231fde780 IN 0.0 80792 2276 zabbix_agentd
1207 1 3 ffff8800bb504500 IN 0.0 110048 756 agetty
1326 1 1 ffff88022e113980 IN 0.0 89476 2092 master
1332 1326 0 ffff880230bda280 IN 0.0 89648 3836 qmgr
1352 1 1 ffff88022e332e00 IN 0.0 124192 1584 crond
1353 1 3 ffff88022e115080 IN 0.0 25672 1820 ntpd
1376 2 2 ffff88022e110b80 IN 0.0 0 0 [kworker/2:1H]
1912 2 0 ffff880144a39700 IN 0.0 0 0 [kworker/0:2]
2065 2 3 ffff88022e322280 IN 0.0 0 0 [kworker/3:1H]
10484 2 1 ffff88014cf45c00 IN 0.0 0 0 [kworker/1:2H]
10833 2 2 ffff88009e142e00 IN 0.0 0 0 [kworker/2:2]
12810 1 3 ffff8800bb331700 IN 46.4 9428784 4377788 java
12811 1 3 ffff8800bb336780 IN 46.4 9428784 4377788 java
30122 2 2 ffff88022fb85080 IN 0.0 0 0 [kworker/u320:0]
41953 2 3 ffff88006e411700 IN 0.0 0 0 [kworker/3:0]
42462 2 2 ffff88014568ae00 IN 0.0 0 0 [kworker/u320:2]
49529 1326 3 ffff88022ff47300 IN 0.0 89580 3940 pickup
50458 1 1 ffff8800bb330000 IN 46.4 9428784 4377788 java
51760 2 1 ffff88022c794500 IN 0.0 0 0 [kworker/1:1]
53165 2 1 ffff88022c792e00 IN 0.0 0 0 [kworker/1:0H]
54670 1 3 ffff8802303ce780 IN 46.4 9428784 4377788 java
55240 1 1 ffff88009e387300 IN 0.0 64948 1084 rpcbind
58290 1 3 ffff88014c99b980 IN 46.4 9428784 4377788 java
58294 1 3 ffff88014c99e780 IN 46.4 9428784 4377788 java
58613 2 0 ffff880036a8dc00 IN 0.0 0 0 [kworker/0:2H]
59013 2 1 ffff8800bb330b80 IN 0.0 0 0 [kworker/1:0]
59267 2 0 ffff8800bb1d5080 IN 0.0 0 0 [kworker/0:1]
59424 2 2 ffff8800bb1d0000 IN 0.0 0 0 [kworker/2:0]
59918 2 3 ffff8800bb1d2280 IN 0.0 0 0 [kworker/3:1]
59931 1 1 ffff88009e10c500 IN 46.4 9428784 4377788 java
122082 1 3 ffff880230e36780 IN 46.4 9428784 4377788 java
122083 1 3 ffff880230e30000 IN 46.4 9428784 4377788 java
122084 1 2 ffff880230e33980 IN 46.4 9428784 4377788 java
122085 1 0 ffff880230e35c00 IN 46.4 9428784 4377788 java
122086 1 1 ffff880230e37300 IN 46.4 9428784 4377788 java
149740 1 2 ffff88009e10b980 IN 2.8 4143428 265568 java
crash>
crash>
crash> dis -l tcp_sendmsg+261
/usr/src/debug/kernel-3.10.0-327.el7/linux-3.10.0-327.el7.x86_64/arch/x86/include/asm/bitops.h: 104
0xffffffff81576a15 <tcp_sendmsg+261>: lock andb $0xfe,0x8(%rax)
crash>
crash> exit
[END] 2019/10/19 14:28:41